Neural Rendering for Sensor Adaptation in 3D Object Detection

TL;DR

This paper investigates the impact of sensor variability on 3D object detection in autonomous vehicles, introduces a new dataset to simulate sensor gaps, and proposes a neural rendering-based adaptation method to improve cross-sensor performance.

Contribution

It introduces CamShift, a dataset for cross-sensor evaluation, and presents a neural rendering pipeline for sensor adaptation in 3D detection.

Findings

Dense BEV architectures like BEVFormer are most robust to sensor changes.

Neural rendering-based adaptation significantly reduces the cross-sensor performance gap.

The proposed method enables dataset reuse across different vehicle sensor setups.

Abstract

Autonomous vehicles often have varying camera sensor setups, which is inevitable due to restricted placement options for different vehicle types. Training a perception model on one particular setup and evaluating it on a new, different sensor setup reveals the so-called cross-sensor domain gap, typically leading to a degradation in accuracy. In this paper, we investigate the impact of the cross-sensor domain gap on state-of-the-art 3D object detectors. To this end, we introduce CamShift, a dataset inspired by nuScenes and created in CARLA to specifically simulate the domain gap between subcompact vehicles and sport utility vehicles (SUVs). Using CamShift, we demonstrate significant cross-sensor performance degradation, identify robustness dependencies on model architecture, and propose a data-driven solution to mitigate the effect. On the one hand, we show that model architectures based on a dense Bird's Eye View (BEV) representation with backward projection, such as BEVFormer, are the most robust against varying sensor configurations. On the other hand, we propose a novel data-driven sensor adaptation pipeline based on neural rendering, which can transform entire datasets to match different camera sensor setups. Applying this approach improves performance across all investigated 3D object detectors, mitigating the cross-sensor domain gap by a large margin and reducing the need for new data collection by enabling efficient data reusability across vehicles with different sensor setups. The CamShift dataset and the sensor adaptation benchmark are available at https://dmholtz.github.io/camshift/.