SF3D-RGB: Scene Flow Estimation from Monocular Camera and Sparse LiDAR

TL;DR

This paper introduces SF3D-RGB, a deep learning model that combines monocular images and sparse LiDAR data for accurate and efficient scene flow estimation in dynamic scenes.

Contribution

The paper presents a novel end-to-end architecture that fuses image and LiDAR features for improved scene flow estimation, outperforming single-modality methods.

Findings

Outperforms single-modality scene flow methods on real-world datasets.

Achieves better accuracy with fewer parameters than existing fusion-based approaches.

Balances accuracy and efficiency effectively.

Abstract

Scene flow estimation is an extremely important task in computer vision to support the perception of dynamic changes in the scene. For robust scene flow, learning-based approaches have recently achieved impressive results using either image-based or LiDAR-based modalities. However, these methods have tended to focus on the use of a single modality. To tackle these problems, we present a deep learning architecture, SF3D-RGB, that enables sparse scene flow estimation using 2D monocular images and 3D point clouds (e.g., acquired by LiDAR) as inputs. Our architecture is an end-to-end model that first encodes information from each modality into features and fuses them together. Then, the fused features enhance a graph matching module for better and more robust mapping matrix computation to generate an initial scene flow. Finally, a residual scene flow module further refines the initial scene flow. Our model is designed to strike a balance between accuracy and efficiency. Furthermore, experiments show that our proposed method outperforms single-modality methods and achieves better scene flow accuracy on real-world datasets while using fewer parameters compared to other state-of-the-art methods with fusion.