FreeGen: Feed-Forward Reconstruction-Generation Co-Training for Free-Viewpoint Driving Scene Synthesis

TL;DR

FreeGen introduces a co-training framework combining reconstruction and generation models to synthesize consistent and realistic free-viewpoint driving scenes, enabling better off-trajectory rendering and state-of-the-art performance.

Contribution

It presents a novel feed-forward co-training approach that jointly optimizes reconstruction and generation models for improved free-viewpoint scene synthesis.

Findings

Achieves state-of-the-art results in free-viewpoint scene synthesis

Ensures interpolation consistency across viewpoints

Enhances realism at unseen viewpoints

Abstract

Closed-loop simulation and scalable pre-training for autonomous driving require synthesizing free-viewpoint driving scenes. However, existing datasets and generative pipelines rarely provide consistent off-trajectory observations, limiting large-scale evaluation and training. While recent generative models demonstrate strong visual realism, they struggle to jointly achieve interpolation consistency and extrapolation realism without per-scene optimization. To address this, we propose FreeGen, a feed-forward reconstruction-generation co-training framework for free-viewpoint driving scene synthesis. The reconstruction model provides stable geometric representations to ensure interpolation consistency, while the generation model performs geometry-aware enhancement to improve realism at unseen viewpoints. Through co-training, generative priors are distilled into the reconstruction model to improve off-trajectory rendering, and the refined geometry in turn offers stronger structural guidance for generation. Experiments demonstrate that FreeGen achieves state-of-the-art performance for free-viewpoint driving scene synthesis.