Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos

TL;DR

This paper presents an unsupervised method for category-level 3D pose estimation from object-centric videos, using a novel multi-view alignment and a dense correspondence model, eliminating the need for human annotations or CAD models.

Contribution

It introduces a two-step pipeline with a multi-view alignment procedure and a dense correspondence model for unsupervised 3D pose estimation from videos.

Findings

Outperforms all baselines in unsupervised video alignment

Provides robust and accurate 3D pose predictions in-the-wild

Enables learning from casual, unannotated videos

Abstract

Category-level 3D pose estimation is a fundamentally important problem in computer vision and robotics, e.g. for embodied agents or to train 3D generative models. However, so far methods that estimate the category-level object pose require either large amounts of human annotations, CAD models or input from RGB-D sensors. In contrast, we tackle the problem of learning to estimate the category-level 3D pose only from casually taken object-centric videos without human supervision. We propose a two-step pipeline: First, we introduce a multi-view alignment procedure that determines canonical camera poses across videos with a novel and robust cyclic distance formulation for geometric and appearance matching using reconstructed coarse meshes and DINOv2 features. In a second step, the canonical poses and reconstructed meshes enable us to train a model for 3D pose estimation from a single image. In particular, our model learns to estimate dense correspondences between images and a prototypical 3D template by predicting, for each pixel in a 2D image, a feature vector of the corresponding vertex in the template mesh. We demonstrate that our method outperforms all baselines at the unsupervised alignment of object-centric videos by a large margin and provides faithful and robust predictions in-the-wild. Our code and data is available at https://github.com/GenIntel/uns-obj-pose3d.