DrivingScene: A Multi-Task Online Feed-Forward 3D Gaussian Splatting Method for Dynamic Driving Scenes

TL;DR

DrivingScene is a real-time, online method that reconstructs dynamic 3D scenes from minimal input, effectively modeling scene motion and outperforming existing approaches in quality and efficiency.

Contribution

It introduces a lightweight residual flow network with a coarse-to-fine training paradigm for high-quality 4D dynamic scene reconstruction from limited views.

Findings

Outperforms state-of-the-art in dynamic scene reconstruction

Generates high-quality depth, scene flow, and 3D point clouds online

Effective in novel view synthesis

Abstract

Real-time, high-fidelity reconstruction of dynamic driving scenes is challenged by complex dynamics and sparse views, with prior methods struggling to balance quality and efficiency. We propose DrivingScene, an online, feed-forward framework that reconstructs 4D dynamic scenes from only two consecutive surround-view images. Our key innovation is a lightweight residual flow network that predicts the non-rigid motion of dynamic objects per camera on top of a learned static scene prior, explicitly modeling dynamics via scene flow. We also introduce a coarse-to-fine training paradigm that circumvents the instabilities common to end-to-end approaches. Experiments on nuScenes dataset show our image-only method simultaneously generates high-quality depth, scene flow, and 3D Gaussian point clouds online, significantly outperforming state-of-the-art methods in both dynamic reconstruction and novel view synthesis.