REOcc: Camera-Radar Fusion with Radar Feature Enrichment for 3D Occupancy Prediction

TL;DR

REOcc is a novel camera-radar fusion network that enriches radar features to improve 3D occupancy prediction, especially in challenging environments, by addressing radar data sparsity and noise.

Contribution

The paper introduces REOcc, a new fusion network with Radar Densifier and Radar Amplifier modules that enhance radar feature quality for better 3D occupancy prediction.

Findings

Significant performance improvements over camera-only models.

Effective mitigation of radar data sparsity and noise.

Enhanced detection of dynamic objects.

Abstract

Vision-based 3D occupancy prediction has made significant advancements, but its reliance on cameras alone struggles in challenging environments. This limitation has driven the adoption of sensor fusion, among which camera-radar fusion stands out as a promising solution due to their complementary strengths. However, the sparsity and noise of the radar data limits its effectiveness, leading to suboptimal fusion performance. In this paper, we propose REOcc, a novel camera-radar fusion network designed to enrich radar feature representations for 3D occupancy prediction. Our approach introduces two main components, a Radar Densifier and a Radar Amplifier, which refine radar features by integrating spatial and contextual information, effectively enhancing spatial density and quality. Extensive experiments on the Occ3D-nuScenes benchmark demonstrate that REOcc achieves significant performance gains over the camera-only baseline model, particularly in dynamic object classes. These results underscore REOcc's capability to mitigate the sparsity and noise of the radar data. Consequently, radar complements camera data more effectively, unlocking the full potential of camera-radar fusion for robust and reliable 3D occupancy prediction.