LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera

TL;DR

This paper introduces LiCROcc, a novel multi-modal fusion framework using radar, LiDAR, and camera data to improve semantic scene completion robustness and accuracy in autonomous driving, especially under adverse weather conditions.

Contribution

It proposes a three-stage BEV fusion architecture and cross-modal distillation modules to enhance radar-only and radar-camera SSC performance, surpassing baseline methods.

Findings

R-LiCROcc achieves 22.9% mIOU improvement over baseline.

RC-LiCROcc surpasses baseline by 44.1% mIOU.

The approach demonstrates robustness against weather and illumination changes.

Abstract

Semantic Scene Completion (SSC) is pivotal in autonomous driving perception, frequently confronted with the complexities of weather and illumination changes. The long-term strategy involves fusing multi-modal information to bolster the system's robustness. Radar, increasingly utilized for 3D target detection, is gradually replacing LiDAR in autonomous driving applications, offering a robust sensing alternative. In this paper, we focus on the potential of 3D radar in semantic scene completion, pioneering cross-modal refinement techniques for improved robustness against weather and illumination changes, and enhancing SSC performance.Regarding model architecture, we propose a three-stage tight fusion approach on BEV to realize a fusion framework for point clouds and images. Based on this foundation, we designed three cross-modal distillation modules-CMRD, BRD, and PDD. Our approach enhances the performance in both radar-only (R-LiCROcc) and radar-camera (RC-LiCROcc) settings by distilling to them the rich semantic and structural information of the fused features of LiDAR and camera. Finally, our LC-Fusion (teacher model), R-LiCROcc and RC-LiCROcc achieve the best performance on the nuScenes-Occupancy dataset, with mIOU exceeding the baseline by 22.9%, 44.1%, and 15.5%, respectively. The project page is available at https://hr-zju.github.io/LiCROcc/.