JOGS: Joint Optimization of Pose Estimation and 3D Gaussian Splatting

TL;DR

JOGS introduces a unified, iterative framework that jointly optimizes camera poses and 3D Gaussian scene representations, eliminating the need for external pose estimation tools and improving reconstruction accuracy.

Contribution

It presents a novel co-optimization strategy that decouples pose and scene parameter updates, enhancing scene reconstruction and pose accuracy without pre-calibrated inputs.

Findings

Outperforms COLMAP-free methods in reconstruction quality

Surpasses COLMAP-based baseline in challenging scenarios

Reduces projection errors in large viewpoint variations

Abstract

Traditional novel view synthesis methods heavily rely on external camera pose estimation tools such as COLMAP, which often introduce computational bottlenecks and propagate errors. To address these challenges, we propose a unified framework that jointly optimizes 3D Gaussian points and camera poses without requiring pre-calibrated inputs. Our approach iteratively refines 3D Gaussian parameters and updates camera poses through a novel co-optimization strategy, ensuring simultaneous improvements in scene reconstruction fidelity and pose estimation accuracy. The key innovation lies in decoupling the joint optimization into two interleaved phases: first, updating 3D Gaussian parameters via differentiable rendering with fixed poses, and second, refining camera poses using a customized 3D optical flow algorithm that incorporates geometric and photometric constraints. This formulation progressively reduces projection errors, particularly in challenging scenarios with large viewpoint variations and sparse feature distributions, where traditional methods struggle. Extensive evaluations on multiple datasets demonstrate that our approach significantly outperforms existing COLMAP-free techniques in reconstruction quality, and also surpasses the standard COLMAP-based baseline in general.