SSRFlow: Semantic-aware Fusion with Spatial Temporal Re-embedding for Real-world Scene Flow

TL;DR

SSRFlow introduces a semantic-aware fusion framework with spatial-temporal re-embedding and domain adaptation, significantly improving scene flow estimation accuracy in real-world LiDAR data by addressing semantic, deformation, and domain gap challenges.

Contribution

The paper proposes a novel dual cross attentive fusion, spatial-temporal re-embedding, and domain adaptive losses to enhance scene flow estimation in real-world scenarios.

Findings

Achieves state-of-the-art performance on multiple datasets.

Outperforms existing methods in real-world LiDAR scene flow estimation.

Effectively bridges synthetic-to-real domain gap.

Abstract

Scene flow, which provides the 3D motion field of the first frame from two consecutive point clouds, is vital for dynamic scene perception. However, contemporary scene flow methods face three major challenges. Firstly, they lack global flow embedding or only consider the context of individual point clouds before embedding, leading to embedded points struggling to perceive the consistent semantic relationship of another frame. To address this issue, we propose a novel approach called Dual Cross Attentive (DCA) for the latent fusion and alignment between two frames based on semantic contexts. This is then integrated into Global Fusion Flow Embedding (GF) to initialize flow embedding based on global correlations in both contextual and Euclidean spaces. Secondly, deformations exist in non-rigid objects after the warping layer, which distorts the spatiotemporal relation between the consecutive frames. For a more precise estimation of residual flow at next-level, the Spatial Temporal Re-embedding (STR) module is devised to update the point sequence features at current-level. Lastly, poor generalization is often observed due to the significant domain gap between synthetic and LiDAR-scanned datasets. We leverage novel domain adaptive losses to effectively bridge the gap of motion inference from synthetic to real-world. Experiments demonstrate that our approach achieves state-of-the-art (SOTA) performance across various datasets, with particularly outstanding results in real-world LiDAR-scanned situations. Our code will be released upon publication.