Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perception

TL;DR

HyDRa introduces a novel camera-radar fusion architecture that enhances 3D perception for autonomous driving by combining features in multiple representations, achieving state-of-the-art results in depth prediction and occupancy estimation.

Contribution

The paper presents a hybrid fusion approach with a Height Association Transformer and Radar-weighted Depth Consistency, improving depth prediction and BEV feature quality in camera-radar systems.

Findings

Achieves 64.2 NDS on nuScenes, surpassing previous methods.

Outperforms all camera-based methods on Occ3D with 3.7 mIoU.

Introduces a new fusion architecture with state-of-the-art results.

Abstract

Low-cost, vision-centric 3D perception systems for autonomous driving have made significant progress in recent years, narrowing the gap to expensive LiDAR-based methods. The primary challenge in becoming a fully reliable alternative lies in robust depth prediction capabilities, as camera-based systems struggle with long detection ranges and adverse lighting and weather conditions. In this work, we introduce HyDRa, a novel camera-radar fusion architecture for diverse 3D perception tasks. Building upon the principles of dense BEV (Bird's Eye View)-based architectures, HyDRa introduces a hybrid fusion approach to combine the strengths of complementary camera and radar features in two distinct representation spaces. Our Height Association Transformer module leverages radar features already in the perspective view to produce more robust and accurate depth predictions. In the BEV, we refine the initial sparse representation by a Radar-weighted Depth Consistency. HyDRa achieves a new state-of-the-art for camera-radar fusion of 64.2 NDS (+1.8) and 58.4 AMOTA (+1.5) on the public nuScenes dataset. Moreover, our new semantically rich and spatially accurate BEV features can be directly converted into a powerful occupancy representation, beating all previous camera-based methods on the Occ3D benchmark by an impressive 3.7 mIoU. Code and models are available at https://github.com/phi-wol/hydra.