UniSplat: Unified Spatio-Temporal Fusion via 3D Latent Scaffolds for Dynamic Driving Scene Reconstruction

TL;DR

UniSplat introduces a unified 3D latent scaffold framework for dynamic scene reconstruction in autonomous driving, effectively handling sparse views and scene dynamics to achieve state-of-the-art results.

Contribution

The paper proposes a novel 3D latent scaffold and fusion mechanism that improves dynamic scene reconstruction and novel view synthesis in complex driving environments.

Findings

Achieves state-of-the-art novel view synthesis performance.

Provides robust reconstructions even outside original camera coverage.

Enables streaming scene completion with persistent static scene memory.

Abstract

Feed-forward 3D reconstruction for autonomous driving has advanced rapidly, yet existing methods struggle with the joint challenges of sparse, non-overlapping camera views and complex scene dynamics. We present UniSplat, a general feed-forward framework that learns robust dynamic scene reconstruction through unified latent spatio-temporal fusion. UniSplat constructs a 3D latent scaffold, a structured representation that captures geometric and semantic scene context by leveraging pretrained foundation models. To effectively integrate information across spatial views and temporal frames, we introduce an efficient fusion mechanism that operates directly within the 3D scaffold, enabling consistent spatio-temporal alignment. To ensure complete and detailed reconstructions, we design a dual-branch decoder that generates dynamic-aware Gaussians from the fused scaffold by combining point-anchored refinement with voxel-based generation, and maintain a persistent memory of static Gaussians to enable streaming scene completion beyond current camera coverage. Extensive experiments on real-world datasets demonstrate that UniSplat achieves state-of-the-art performance in novel view synthesis, while providing robust and high-quality renderings even for viewpoints outside the original camera coverage.