Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation

TL;DR

This paper introduces Radar-Modulated Selection (RMS), a novel method for radar-camera depth estimation that integrates radar within the model's core, leading to state-of-the-art accuracy and efficiency.

Contribution

The paper proposes RMS, a new approach that modulates the model internally with radar data, enabling linear-cost cross-modal coupling and better fallback to image-only processing.

Findings

Achieves state-of-the-art depth estimation on nuScenes with 34% MAE reduction.

RMS provides lowest single-frame latency at 26.8ms.

In-scan selection replaces out-of-scan fusion without loss of accuracy.

Abstract

Radar-camera depth estimation must turn an ultra-sparse, all-weather, metric radar signal into a dense per-pixel depth map. Existing methods -- concatenation, confidence-aware gating, sparse supervision, graph-based extraction -- combine radar and image features outside the backbone's sequence operator, and even cross-modal Mamba variants leave the selection mechanism itself unimodal. We argue that the selection mechanism is the right place for radar to enter. We introduce Radar-Modulated Selection (RMS), a minimal and principled way to inject radar into Mamba's selective scan: radar modulates the scan from within, adding zero-initialised perturbations to the step size $\Delta$ and readout $\mathbf{C}$ while leaving the input projection $\mathbf{B}$ and state dynamics $\mathbf{A}$ image-only. The construction is exactly equivalent to a pretrained image-only Mamba at initialisation, ensuring radar only influences the model where it improves accuracy. Two further properties follow that out-of-scan fusion cannot offer: linear-cost cross-modal coupling at every recurrence step, and a natural fallback to the image-only backbone when radar is absent. We deploy RMS in a Multi-View Scan Pyramid (MVSP) that matches the fusion operator to radar's spatial reach at each scale. SemoDepth achieves state-of-the-art performance on nuScenes, reducing MAE by 34.0%, 29.9%, and 29.9% over the previous best at 0--50, 0--70, and 0--80m, while attaining the lowest single-frame latency (26.8ms). A further ablation shows that out-of-scan feature blending adds no accuracy on top of RMS, providing empirical validation that in-scan selection can replace out-of-scan fusion.