Radar-Camera BEV Multi-Task Learning with Cross-Task Attention Bridge for Joint 3D Detection and Segmentation

TL;DR

This paper introduces CTAB, a cross-task attention module that enhances joint 3D detection and segmentation in BEV space by sharing features between tasks, improving performance on nuScenes.

Contribution

The paper presents a novel bidirectional attention bridge for multi-task BEV perception, enabling effective feature sharing between detection and segmentation tasks.

Findings

CTAB improves segmentation accuracy on nuScenes for 7 classes.

The joint model achieves comparable mIoU on 4 classes while providing 3D detection.

The framework enhances BEV representations with multi-scale deformable attention.

Abstract

Bird's-eye-view (BEV) representations are the dominant paradigm for 3D perception in autonomous driving, providing a unified spatial canvas where detection and segmentation features are geometrically registered to the same physical coordinate system. However, existing radar-camera fusion methods treat these tasks in isolation, missing the opportunity to share complementary information between them: detection features encode object-level geometry that can sharpen segmentation boundaries, while segmentation features provide dense semantic context that can anchor detection. We propose \textbf{CTAB} (Cross-Task Attention Bridge), a bidirectional module that exchanges features between detection and segmentation branches via multi-scale deformable attention in shared BEV space. CTAB is integrated into a multi-task framework with an Instance Normalization-based segmentation decoder and learnable BEV upsampling to provide a more detailed BEV representation. On nuScenes, CTAB improves segmentation on 7 classes over the joint multi-task baseline at essentially neutral detection. On a 4-class subset (drivable area, pedestrian crossing, walkway, vehicle), our joint multi-task model reaches comparable mIoU on 4 classes while simultaneously providing 3D detection.