PreF3R: Pose-Free Feed-Forward 3D Gaussian Splatting from Variable-length Image Sequence

TL;DR

PreF3R is a real-time, pose-free 3D reconstruction method from image sequences that enables efficient novel-view rendering without camera calibration, using a dense Gaussian parameter prediction and a spatial memory network.

Contribution

It introduces a novel pose-free, feed-forward 3D reconstruction approach that extends pair-wise 3D structure reconstruction to sequential inputs with a spatial memory network.

Findings

Reconstructs 3D Gaussian field at 20 FPS in real-time.

Effective for pose-free, novel-view synthesis.

Shows robust generalization to unseen scenes.

Abstract

We present PreF3R, Pose-Free Feed-forward 3D Reconstruction from an image sequence of variable length. Unlike previous approaches, PreF3R removes the need for camera calibration and reconstructs the 3D Gaussian field within a canonical coordinate frame directly from a sequence of unposed images, enabling efficient novel-view rendering. We leverage DUSt3R's ability for pair-wise 3D structure reconstruction, and extend it to sequential multi-view input via a spatial memory network, eliminating the need for optimization-based global alignment. Additionally, PreF3R incorporates a dense Gaussian parameter prediction head, which enables subsequent novel-view synthesis with differentiable rasterization. This allows supervising our model with the combination of photometric loss and pointmap regression loss, enhancing both photorealism and structural accuracy. Given a sequence of ordered images, PreF3R incrementally reconstructs the 3D Gaussian field at 20 FPS, therefore enabling real-time novel-view rendering. Empirical experiments demonstrate that PreF3R is an effective solution for the challenging task of pose-free feed-forward novel-view synthesis, while also exhibiting robust generalization to unseen scenes.