VISC: mmWave Radar Scene Flow Estimation using Pervasive Visual-Inertial Supervision

TL;DR

This paper introduces VISC, a novel mmWave radar scene flow estimation framework supervised by visual-inertial data, enabling cost-effective, crowdsourced training for autonomous vehicles, and outperforming LiDAR-based methods in challenging environments.

Contribution

It presents a drift-free rigid transformation estimator and an optical-mmWave supervision extraction module, enhancing scene flow estimation without relying on expensive LiDAR data.

Findings

Outperforms LiDAR-based methods in smoke-filled environments

Enables crowdsourced training using visual-inertial data

Improves static and dynamic scene flow estimation

Abstract

This work proposes a mmWave radar's scene flow estimation framework supervised by data from a widespread visual-inertial (VI) sensor suite, allowing crowdsourced training data from smart vehicles. Current scene flow estimation methods for mmWave radar are typically supervised by dense point clouds from 3D LiDARs, which are expensive and not widely available in smart vehicles. While VI data are more accessible, visual images alone cannot capture the 3D motions of moving objects, making it difficult to supervise their scene flow. Moreover, the temporal drift of VI rigid transformation also degenerates the scene flow estimation of static points. To address these challenges, we propose a drift-free rigid transformation estimator that fuses kinematic model-based ego-motions with neural network-learned results. It provides strong supervision signals to radar-based rigid transformation and infers the scene flow of static points. Then, we develop an optical-mmWave supervision extraction module that extracts the supervision signals of radar rigid transformation and scene flow. It strengthens the supervision by learning the scene flow of dynamic points with the joint constraints of optical and mmWave radar measurements. Extensive experiments demonstrate that, in smoke-filled environments, our method even outperforms state-of-the-art (SOTA) approaches using costly LiDARs.