DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving

TL;DR

DrivingRecon is a novel large 4D Gaussian model that efficiently reconstructs street scenes for autonomous driving, enabling real-time applications and improving scene quality over existing offline methods.

Contribution

The paper introduces DrivingRecon, a generalizable 4D Gaussian reconstruction model with novel PD-Block and decoupling techniques for better scene reconstruction and view synthesis.

Findings

Significantly improves scene reconstruction quality

Enhances novel view synthesis performance

Enables applications in scene editing and model pre-training

Abstract

Photorealistic 4D reconstruction of street scenes is essential for developing real-world simulators in autonomous driving. However, most existing methods perform this task offline and rely on time-consuming iterative processes, limiting their practical applications. To this end, we introduce the Large 4D Gaussian Reconstruction Model (DrivingRecon), a generalizable driving scene reconstruction model, which directly predicts 4D Gaussian from surround view videos. To better integrate the surround-view images, the Prune and Dilate Block (PD-Block) is proposed to eliminate overlapping Gaussian points between adjacent views and remove redundant background points. To enhance cross-temporal information, dynamic and static decoupling is tailored to better learn geometry and motion features. Experimental results demonstrate that DrivingRecon significantly improves scene reconstruction quality and novel view synthesis compared to existing methods. Furthermore, we explore applications of DrivingRecon in model pre-training, vehicle adaptation, and scene editing. Our code is available at https://github.com/EnVision-Research/DriveRecon.