Camera simulation for robot simulation: how important are various camera model components?

TL;DR

This paper investigates how various components of camera models influence the realism of synthetic images in robotic simulation, highlighting the relative importance of scene effects versus local artifacts.

Contribution

It provides a quantitative analysis of the impact of different camera model components on the sim-to-real gap in robotics simulations.

Findings

Scene effects like lighting and reflection have greater impact than local artifacts.

Lens distortion and signal processing significantly affect scene-level realism.

Large-scale changes in scene parameters outweigh local feature artifacts.

Abstract

Modeling cameras for the simulation of autonomous robotics is critical for generating synthetic images with appropriate realism to effectively evaluate a perception algorithm in simulation. In many cases though, simulated images are produced by traditional rendering techniques that exclude or superficially handle processing steps and aspects encountered in the actual camera pipeline. The purpose of this contribution is to quantify the degree to which the exclusion from the camera model of various image generation steps or aspects affect the sim-to-real gap in robotics. We investigate what happens if one ignores aspects tied to processes from within the physical camera, e.g., lens distortion, noise, and signal processing; scene effects, e.g., lighting and reflection; and rendering quality. The results of the study demonstrate, quantitatively, that large-scale changes to color, scene, and location have far greater impact than model aspects concerned with local, feature-level artifacts. Moreover, we show that these scene-level aspects can stem from lens distortion and signal processing, particularly when considering white-balance and auto-exposure modeling.