Cameras as Rays: Pose Estimation via Ray Diffusion

TL;DR

This paper introduces a novel distributed ray-based representation for camera pose estimation from sparse views, leveraging transformers and diffusion models to improve accuracy and uncertainty modeling, achieving state-of-the-art results.

Contribution

It proposes a new ray-based distributed representation for camera pose estimation, combined with transformer and diffusion models, advancing accuracy and generalization in sparse-view scenarios.

Findings

State-of-the-art performance on CO3D dataset

Effective generalization to unseen categories

Improved pose precision with ray-based representation

Abstract

Estimating camera poses is a fundamental task for 3D reconstruction and remains challenging given sparsely sampled views (<10). In contrast to existing approaches that pursue top-down prediction of global parametrizations of camera extrinsics, we propose a distributed representation of camera pose that treats a camera as a bundle of rays. This representation allows for a tight coupling with spatial image features improving pose precision. We observe that this representation is naturally suited for set-level transformers and develop a regression-based approach that maps image patches to corresponding rays. To capture the inherent uncertainties in sparse-view pose inference, we adapt this approach to learn a denoising diffusion model which allows us to sample plausible modes while improving performance. Our proposed methods, both regression- and diffusion-based, demonstrate state-of-the-art performance on camera pose estimation on CO3D while generalizing to unseen object categories and in-the-wild captures.