FreeScale: Scaling 3D Scenes via Certainty-Aware Free-View Generation

TL;DR

FreeScale introduces a scene reconstruction-based data augmentation framework that improves novel view synthesis models by generating high-quality, semantically meaningful training data from limited real-world sequences.

Contribution

It proposes a certainty-aware sampling strategy to effectively utilize imperfect scene reconstructions for scalable, high-quality training data generation in 3D vision tasks.

Findings

Achieved a 2.7 dB PSNR improvement on out-of-distribution benchmarks.

Enhanced 3D Gaussian Splatting optimization with generated data.

Provided a practical data generation engine to address training data scarcity.

Abstract

The development of generalizable Novel View Synthesis (NVS) models is critically limited by the scarcity of large-scale training data featuring diverse and precise camera trajectories. While real-world captures are photorealistic, they are typically sparse and discrete. Conversely, synthetic data scales but suffers from a domain gap and often lacks realistic semantics. We introduce FreeScale, a novel framework that leverages the power of scene reconstruction to transform limited real-world image sequences into a scalable source of high-quality training data. Our key insight is that an imperfect reconstructed scene serves as a rich geometric proxy, but naively sampling from it amplifies artifacts. To this end, we propose a certainty-aware free-view sampling strategy identifying novel viewpoints that are both semantically meaningful and minimally affected by reconstruction errors. We demonstrate FreeScale's effectiveness by scaling up the training of feedforward NVS models, achieving a notable gain of 2.7 dB in PSNR on challenging out-of-distribution benchmarks. Furthermore, we show that the generated data can actively enhance per-scene 3D Gaussian Splatting optimization, leading to consistent improvements across multiple datasets. Our work provides a practical and powerful data generation engine to overcome a fundamental bottleneck in 3D vision. Project page: https://mvp-ai-lab.github.io/FreeScale.