Neural-Sim: Learning to Generate Training Data with NeRF

TL;DR

Neural-Sim introduces a fully differentiable NeRF-based pipeline that generates targeted synthetic training data automatically, improving object detection performance without manual scene tuning.

Contribution

It is the first to integrate NeRFs in a closed-loop with task loss for automatic, optimized synthetic data generation tailored to specific applications.

Findings

Effective in synthetic and real-world object detection tasks

Creates a new 'YCB-in-the-Wild' dataset for benchmarking

Reduces manual effort in data generation process

Abstract

Training computer vision models usually requires collecting and labeling vast amounts of imagery under a diverse set of scene configurations and properties. This process is incredibly time-consuming, and it is challenging to ensure that the captured data distribution maps well to the target domain of an application scenario. Recently, synthetic data has emerged as a way to address both of these issues. However, existing approaches either require human experts to manually tune each scene property or use automatic methods that provide little to no control; this requires rendering large amounts of random data variations, which is slow and is often suboptimal for the target domain. We present the first fully differentiable synthetic data pipeline that uses Neural Radiance Fields (NeRFs) in a closed-loop with a target application's loss function. Our approach generates data on-demand, with no human labor, to maximize accuracy for a target task. We illustrate the effectiveness of our method on synthetic and real-world object detection tasks. We also introduce a new "YCB-in-the-Wild" dataset and benchmark that provides a test scenario for object detection with varied poses in real-world environments.