Detailed Geometry and Appearance from Opportunistic Motion

TL;DR

This paper introduces a method to improve 3D reconstruction from sparse fixed cameras by exploiting object motion to simulate additional viewpoints, enabling more accurate geometry and appearance recovery.

Contribution

It presents a joint pose and shape optimization framework using 2D Gaussian splatting and a novel appearance model that separates diffuse and specular components.

Findings

Achieves more accurate geometry and appearance reconstruction than existing methods.

Effectively leverages object motion to overcome viewpoint limitations.

Demonstrates robustness on synthetic and real-world datasets with sparse viewpoints.

Abstract

Reconstructing 3D geometry and appearance from a sparse set of fixed cameras is a foundational task with broad applications, yet it remains fundamentally constrained by the limited viewpoints. We show that this bound can be broken by exploiting opportunistic object motion: as a person manipulates an object~(e.g., moving a chair or lifting a mug), the static cameras effectively ``orbit'' the object in its local coordinate frame, providing additional virtual viewpoints. Harnessing this object motion, however, poses two challenges: the tight coupling of object pose and geometry estimation and the complex appearance variations of a moving object under static illumination. We address these by formulating a joint pose and shape optimization using 2D Gaussian splatting with alternating minimization of 6DoF trajectories and primitive parameters, and by introducing a novel appearance model that factorizes diffuse and specular components with reflected directional probing within the spherical harmonics space. Extensive experiments on synthetic and real-world datasets with extremely sparse viewpoints demonstrate that our method recovers significantly more accurate geometry and appearance than state-of-the-art baselines.