RaLiFlow: Scene Flow Estimation with 4D Radar and LiDAR Point Clouds

TL;DR

RaLiFlow introduces a novel framework for scene flow estimation using combined 4D radar and LiDAR data, addressing the lack of datasets and fusion methods for these modalities, and demonstrating superior performance over single-modal approaches.

Contribution

It presents the first joint scene flow learning framework for radar and LiDAR, along with a new dataset and a dynamic-aware fusion module for improved multimodal scene flow estimation.

Findings

Outperforms existing LiDAR-based and radar-based methods significantly

Introduces a new Radar-LiDAR scene flow dataset and preprocessing strategy

Effective radar-LiDAR fusion via the DBCF module enhances dynamic scene understanding

Abstract

Recent multimodal fusion methods, integrating images with LiDAR point clouds, have shown promise in scene flow estimation. However, the fusion of 4D millimeter wave radar and LiDAR remains unexplored. Unlike LiDAR, radar is cheaper, more robust in various weather conditions and can detect point-wise velocity, making it a valuable complement to LiDAR. However, radar inputs pose challenges due to noise, low resolution, and sparsity. Moreover, there is currently no dataset that combines LiDAR and radar data specifically for scene flow estimation. To address this gap, we construct a Radar-LiDAR scene flow dataset based on a public real-world automotive dataset. We propose an effective preprocessing strategy for radar denoising and scene flow label generation, deriving more reliable flow ground truth for radar points out of the object boundaries. Additionally, we introduce RaLiFlow, the first joint scene flow learning framework for 4D radar and LiDAR, which achieves effective radar-LiDAR fusion through a novel Dynamic-aware Bidirectional Cross-modal Fusion (DBCF) module and a carefully designed set of loss functions. The DBCF module integrates dynamic cues from radar into the local cross-attention mechanism, enabling the propagation of contextual information across modalities. Meanwhile, the proposed loss functions mitigate the adverse effects of unreliable radar data during training and enhance the instance-level consistency in scene flow predictions from both modalities, particularly for dynamic foreground areas. Extensive experiments on the repurposed scene flow dataset demonstrate that our method outperforms existing LiDAR-based and radar-based single-modal methods by a significant margin.