DualPM: Dual Posed-Canonical Point Maps for 3D Shape and Pose Reconstruction

TL;DR

This paper introduces Dual Point Maps (DualPM), a novel representation for 3D shape and pose reconstruction of deformable objects, enabling effective training on synthetic data and generalization to real images.

Contribution

The paper proposes DualPM, a new dual point map representation for 3D shape and pose estimation, extending to amodal reconstruction and demonstrating strong generalization from synthetic to real data.

Findings

DualPM effectively predicts 3D shape and pose from images.

The approach generalizes well from synthetic to real images.

Significant improvements over previous methods in 3D analysis of quadrupeds.

Abstract

The choice of data representation is a key factor in the success of deep learning in geometric tasks. For instance, DUSt3R recently introduced the concept of viewpoint-invariant point maps, generalizing depth prediction and showing that all key problems in the 3D reconstruction of static scenes can be reduced to predicting such point maps. In this paper, we develop an analogous concept for a very different problem: the reconstruction of the 3D shape and pose of deformable objects. To this end, we introduce Dual Point Maps (DualPM), where a pair of point maps is extracted from the same image-one associating pixels to their 3D locations on the object and the other to a canonical version of the object in its rest pose. We also extend point maps to amodal reconstruction to recover the complete shape of the object, even through self-occlusions. We show that 3D reconstruction and 3D pose estimation can be reduced to the prediction of DualPMs. Empirically, we demonstrate that this representation is a suitable target for deep networks to predict. Specifically, we focus on modeling quadrupeds, showing that DualPMs can be trained purely on synthetic 3D data, consisting of one or two models per category, while generalizing effectively to real images. With this approach, we achieve significant improvements over previous methods for the 3D analysis and reconstruction of such objects.