YoNoSplat: You Only Need One Model for Feedforward 3D Gaussian Splatting

TL;DR

YoNoSplat is a versatile, fast feedforward model capable of reconstructing high-quality 3D Gaussian Splatting representations from various image inputs, including unposed and uncalibrated images, using a novel training strategy.

Contribution

It introduces a novel mixing training strategy and camera normalization techniques, enabling a single model to handle pose-free and pose-dependent 3D scene reconstruction tasks.

Findings

Reconstructs scenes from 100 views in 2.69 seconds on GPU.

Achieves state-of-the-art results on standard benchmarks.

Operates effectively with both posed and unposed, calibrated and uncalibrated inputs.

Abstract

Fast and flexible 3D scene reconstruction from unstructured image collections remains a significant challenge. We present YoNoSplat, a feedforward model that reconstructs high-quality 3D Gaussian Splatting representations from an arbitrary number of images. Our model is highly versatile, operating effectively with both posed and unposed, calibrated and uncalibrated inputs. YoNoSplat predicts local Gaussians and camera poses for each view, which are aggregated into a global representation using either predicted or provided poses. To overcome the inherent difficulty of jointly learning 3D Gaussians and camera parameters, we introduce a novel mixing training strategy. This approach mitigates the entanglement between the two tasks by initially using ground-truth poses to aggregate local Gaussians and gradually transitioning to a mix of predicted and ground-truth poses, which prevents both training instability and exposure bias. We further resolve the scale ambiguity problem by a novel pairwise camera-distance normalization scheme and by embedding camera intrinsics into the network. Moreover, YoNoSplat also predicts intrinsic parameters, making it feasible for uncalibrated inputs. YoNoSplat demonstrates exceptional efficiency, reconstructing a scene from 100 views (at 280x518 resolution) in just 2.69 seconds on an NVIDIA GH200 GPU. It achieves state-of-the-art performance on standard benchmarks in both pose-free and pose-dependent settings. Our project page is at https://botaoye.github.io/yonosplat/.