Learning Scene Dynamics from Point Cloud Sequences

TL;DR

This paper introduces a new approach for estimating 3D scene flow from sequences of point clouds, improving accuracy through recurrent processing and providing a new benchmark for multi-frame scene flow and forecasting.

Contribution

The paper presents SPCM-Net, a novel architecture for sequential scene flow estimation and forecasting from point cloud sequences, along with a new benchmark dataset for evaluation.

Findings

Recurrent processing significantly improves scene flow estimation accuracy.

The proposed method effectively extends to point cloud forecasting.

A new benchmark dataset for multi-frame scene flow and prediction is introduced.

Abstract

Understanding 3D scenes is a critical prerequisite for autonomous agents. Recently, LiDAR and other sensors have made large amounts of data available in the form of temporal sequences of point cloud frames. In this work, we propose a novel problem -- sequential scene flow estimation (SSFE) -- that aims to predict 3D scene flow for all pairs of point clouds in a given sequence. This is unlike the previously studied problem of scene flow estimation which focuses on two frames. We introduce the SPCM-Net architecture, which solves this problem by computing multi-scale spatiotemporal correlations between neighboring point clouds and then aggregating the correlation across time with an order-invariant recurrent unit. Our experimental evaluation confirms that recurrent processing of point cloud sequences results in significantly better SSFE compared to using only two frames. Additionally, we demonstrate that this approach can be effectively modified for sequential point cloud forecasting (SPF), a related problem that demands forecasting future point cloud frames. Our experimental results are evaluated using a new benchmark for both SSFE and SPF consisting of synthetic and real datasets. Previously, datasets for scene flow estimation have been limited to two frames. We provide non-trivial extensions to these datasets for multi-frame estimation and prediction. Due to the difficulty of obtaining ground truth motion for real-world datasets, we use self-supervised training and evaluation metrics. We believe that this benchmark will be pivotal to future research in this area. All code for benchmark and models will be made accessible.