BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation

TL;DR

The BEHAVIOR Vision Suite (BVS) provides a customizable synthetic data generation platform for systematic evaluation and training of computer vision models, addressing limitations of existing synthetic datasets in quality and diversity.

Contribution

BVS introduces a flexible toolset with adjustable parameters for scene, object, and camera configurations, enabling controlled experiments and improved model evaluation.

Findings

Enables systematic robustness testing across domain shifts.

Supports evaluation of scene understanding models.

Facilitates simulation-to-real transfer for vision tasks.

Abstract

The systematic evaluation and understanding of computer vision models under varying conditions require large amounts of data with comprehensive and customized labels, which real-world vision datasets rarely satisfy. While current synthetic data generators offer a promising alternative, particularly for embodied AI tasks, they often fall short for computer vision tasks due to low asset and rendering quality, limited diversity, and unrealistic physical properties. We introduce the BEHAVIOR Vision Suite (BVS), a set of tools and assets to generate fully customized synthetic data for systematic evaluation of computer vision models, based on the newly developed embodied AI benchmark, BEHAVIOR-1K. BVS supports a large number of adjustable parameters at the scene level (e.g., lighting, object placement), the object level (e.g., joint configuration, attributes such as "filled" and "folded"), and the camera level (e.g., field of view, focal length). Researchers can arbitrarily vary these parameters during data generation to perform controlled experiments. We showcase three example application scenarios: systematically evaluating the robustness of models across different continuous axes of domain shift, evaluating scene understanding models on the same set of images, and training and evaluating simulation-to-real transfer for a novel vision task: unary and binary state prediction. Project website: https://behavior-vision-suite.github.io/