Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs

TL;DR

Splatt3R is a novel zero-shot, pose-free 3D reconstruction method from uncalibrated stereo images that uses Gaussian splats for real-time view synthesis, extending a foundation model for better generalization.

Contribution

We introduce Splatt3R, a new method that reconstructs 3D scenes and synthesizes views from uncalibrated images without camera parameters, building upon and extending MASt3R.

Findings

Achieves real-time reconstruction at 4FPS at 512x512 resolution.

Demonstrates strong generalization to in-the-wild images.

Outperforms existing methods in novel view synthesis from uncalibrated stereo pairs.

Abstract

In this paper, we introduce Splatt3R, a pose-free, feed-forward method for in-the-wild 3D reconstruction and novel view synthesis from stereo pairs. Given uncalibrated natural images, Splatt3R can predict 3D Gaussian Splats without requiring any camera parameters or depth information. For generalizability, we build Splatt3R upon a ``foundation'' 3D geometry reconstruction method, MASt3R, by extending it to deal with both 3D structure and appearance. Specifically, unlike the original MASt3R which reconstructs only 3D point clouds, we predict the additional Gaussian attributes required to construct a Gaussian primitive for each point. Hence, unlike other novel view synthesis methods, Splatt3R is first trained by optimizing the 3D point cloud's geometry loss, and then a novel view synthesis objective. By doing this, we avoid the local minima present in training 3D Gaussian Splats from stereo views. We also propose a novel loss masking strategy that we empirically find is critical for strong performance on extrapolated viewpoints. We train Splatt3R on the ScanNet++ dataset and demonstrate excellent generalisation to uncalibrated, in-the-wild images. Splatt3R can reconstruct scenes at 4FPS at 512 x 512 resolution, and the resultant splats can be rendered in real-time.