4DRC-OCC: Robust Semantic Occupancy Prediction Through Fusion of 4D Radar and Camera

TL;DR

This paper introduces a novel fusion of 4D radar and camera data for robust 3D semantic occupancy prediction in autonomous driving, especially under adverse conditions, and provides a new automatically labeled dataset for training.

Contribution

It is the first to combine 4D radar and camera data for semantic occupancy prediction and introduces an automatic labeling method to reduce manual annotation efforts.

Findings

Fusion improves robustness in challenging weather conditions.

Depth cues from cameras enhance 3D scene reconstruction.

The dataset enables effective training with less manual labeling.

Abstract

Autonomous driving requires robust perception across diverse environmental conditions, yet 3D semantic occupancy prediction remains challenging under adverse weather and lighting. In this work, we present the first study combining 4D radar and camera data for 3D semantic occupancy prediction. Our fusion leverages the complementary strengths of both modalities: 4D radar provides reliable range, velocity, and angle measurements in challenging conditions, while cameras contribute rich semantic and texture information. We further show that integrating depth cues from camera pixels enables lifting 2D images to 3D, improving scene reconstruction accuracy. Additionally, we introduce a fully automatically labeled dataset for training semantic occupancy models, substantially reducing reliance on costly manual annotation. Experiments demonstrate the robustness of 4D radar across diverse scenarios, highlighting its potential to advance autonomous vehicle perception.