Towards Physically Consistent 4D Scene Reconstruction for Closed-loop Autonomous Driving Simulation

TL;DR

This paper introduces a novel hierarchical training framework with temporal regularization to achieve physically consistent 4D scene reconstruction, enhancing closed-loop autonomous driving simulation.

Contribution

It proposes Orthogonal Projected Gradient (OPG) and a temporal regularization strategy to improve spatial-temporal disentanglement and scene consistency in 4D reconstructions.

Findings

Maintains stable novel-view synthesis in 4D scene reconstruction.

Outperforms existing methods in observation-reproducing metrics.

Ensures physically consistent scene evolution in closed-loop simulation.

Abstract

High-fidelity street scene reconstruction is pivotal for end-to-end autonomous driving simulation, where novel-view synthesis (NVS) and time-varying information modeling are two fundamental capabilities to facilitate closed-loop training. However, existing 3DGS methods and their 4D extensions fail to simultaneously achieve both. To bridge this gap, we establish an information-geometric diagnostic framework, revealing that this limitation stems from a credit assignment dilemma between spatial and temporal parameters. Specifically, the deterministic coupling between viewpoint and time in single-source observation creates a low-rank structure that induces massive null-space ambiguity between static view-dependent and dynamic time-varying components. Temporal information overshadows spatial cues, causing the estimation variance of spatial parameters to diverge. To address this issue, we propose Orthogonal Projected Gradient (OPG), a hierarchical training method designed to restore spatial identifiability. OPG prioritizes the integrity of spatial representations by securing them in an initial stage, then restricts temporal updates to the spatial null space, enabling proactive credit assignment. While OPG isolates temporal updates algebraically, Temporal Regularization Strategy is proposed to further refine the temporal solution space by imposing a smoothness constraint based on the physical prior of consistent appearance evolution, ensuring that the reconstructed scene remains physically consistent in closed-loop simulation. Extensive experiments demonstrate that our method not only maintains stable NVS capabilities but also demonstrates superior performance in traditional observation-reproducing metrics, which indirectly reflect the capability of modeling temporal dynamics.