Photo-realistic Neural Domain Randomization

TL;DR

This paper introduces Photo-realistic Neural Domain Randomization (PNDR), a neural rendering approach that generates high-quality, photo-realistic images from scene geometry, improving synthetic-to-real transfer in tasks like object detection and depth estimation.

Contribution

The paper presents a modular neural rendering pipeline for neural domain randomization that produces photo-realistic images efficiently, enhancing transfer learning for real-world applications.

Findings

PNDR outperforms state-of-the-art methods in real-world transfer.

It enables efficient, online generation of photo-realistic augmentations.

The approach generalizes well to novel scenes.

Abstract

Synthetic data is a scalable alternative to manual supervision, but it requires overcoming the sim-to-real domain gap. This discrepancy between virtual and real worlds is addressed by two seemingly opposed approaches: improving the realism of simulation or foregoing realism entirely via domain randomization. In this paper, we show that the recent progress in neural rendering enables a new unified approach we call Photo-realistic Neural Domain Randomization (PNDR). We propose to learn a composition of neural networks that acts as a physics-based ray tracer generating high-quality renderings from scene geometry alone. Our approach is modular, composed of different neural networks for materials, lighting, and rendering, thus enabling randomization of different key image generation components in a differentiable pipeline. Once trained, our method can be combined with other methods and used to generate photo-realistic image augmentations online and significantly more efficiently than via traditional ray-tracing. We demonstrate the usefulness of PNDR through two downstream tasks: 6D object detection and monocular depth estimation. Our experiments show that training with PNDR enables generalization to novel scenes and significantly outperforms the state of the art in terms of real-world transfer.