Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization

TL;DR

This paper introduces a method for novel view synthesis using Gaussian Splatting that jointly optimizes camera poses and scene geometry without requiring accurate pose initialization, enabling faster and more robust 3D scene reconstruction.

Contribution

It extends 3D Gaussian Splatting by integrating camera pose optimization with respect to photometric residuals, allowing pose estimation and scene reconstruction without prior pose accuracy.

Findings

Achieves rapid convergence and high accuracy in pose estimation on real-world scenes.

Enables fast joint optimization of geometry and camera poses for scene reconstruction.

Reduces runtime by two to four times compared to existing methods while maintaining state-of-the-art quality.

Abstract

3D Gaussian Splatting has recently emerged as a powerful tool for fast and accurate novel-view synthesis from a set of posed input images. However, like most novel-view synthesis approaches, it relies on accurate camera pose information, limiting its applicability in real-world scenarios where acquiring accurate camera poses can be challenging or even impossible. We propose an extension to the 3D Gaussian Splatting framework by optimizing the extrinsic camera parameters with respect to photometric residuals. We derive the analytical gradients and integrate their computation with the existing high-performance CUDA implementation. This enables downstream tasks such as 6-DoF camera pose estimation as well as joint reconstruction and camera refinement. In particular, we achieve rapid convergence and high accuracy for pose estimation on real-world scenes. Our method enables fast reconstruction of 3D scenes without requiring accurate pose information by jointly optimizing geometry and camera poses, while achieving state-of-the-art results in novel-view synthesis. Our approach is considerably faster to optimize than most competing methods, and several times faster in rendering. We show results on real-world scenes and complex trajectories through simulated environments, achieving state-of-the-art results on LLFF while reducing runtime by two to four times compared to the most efficient competing method. Source code will be available at https://github.com/Schmiddo/noposegs .