Benchmarking Efficient & Effective Camera Pose Estimation Strategies for Novel View Synthesis

TL;DR

This paper introduces a benchmark for camera pose estimation in novel view synthesis, comparing classical and neural network-based SfM methods, and demonstrates strategies to balance efficiency and accuracy.

Contribution

It develops a benchmark for evaluating SfM methods in NVS, highlighting simple feature reduction and hybrid approaches as effective strategies.

Findings

Fewer features speed up classical SfM with high accuracy.

Feed-forward network estimates refined by classical SfM offer optimal efficiency and effectiveness.

Hybrid methods outperform purely neural or classical approaches.

Abstract

Novel view synthesis (NVS) approaches such as NeRFs or 3DGS can produce photo-realistic 3D scene representation from a set of images with known extrinsic and intrinsic parameters. The necessary camera poses and calibrations are typically obtained from the images via Structure-from-Motion (SfM). Classical SfM approaches rely on local feature matches between the images to estimate both the poses and a sparse 3D model of the scene, using bundle adjustment to refine initial pose, intrinsics, and geometry estimates. In order to increase run-time efficiency, recent SfM systems forgo optimization via bundle adjustment. Instead, they train feed-forward (transformer-based) neural networks to directly regress camera parameters and the 3D structure. While orders of magnitude more efficient, such recent works produce significantly less accurate estimates. To stimulate research on developing SfM approaches that are both efficient \emph{and} effective, this paper develops a benchmark focused on SfM for novel view synthesis. Using existing datasets and two simple strategies for making the reconstruction process more efficient, we show that: (1) simply using fewer features already significantly accelerates classical SfM methods while maintaining high pose accuracy. (2) using feed-forward networks to obtain initial estimates and refining them using classical SfM techniques leads to the best efficiency-effectiveness trade-off. We will make our benchmark and code publicly available.