Unsupervised Monocular 3D Keypoint Discovery from Multi-View Diffusion Priors

TL;DR

KeyDiff3D is an unsupervised framework that estimates 3D keypoints from a single image by leveraging multi-view diffusion priors, eliminating the need for manual annotations or multi-view data.

Contribution

It introduces a novel method that uses a pretrained multi-view diffusion model to generate multi-view cues from a single image for 3D keypoint estimation.

Findings

Achieves accurate 3D keypoints on diverse datasets.

Demonstrates strong generalization to in-the-wild images.

Enables manipulation of 3D objects from a single image.

Abstract

This paper introduces KeyDiff3D, a framework for unsupervised monocular 3D keypoints estimation that accurately predicts 3D keypoints from a single image. While previous methods rely on manual annotations or calibrated multi-view images, both of which are expensive to collect, our method enables monocular 3D keypoints estimation using only a collection of single-view images. To achieve this, we leverage powerful geometric priors embedded in a pretrained multi-view diffusion model. In our framework, this model generates multi-view images from a single image, serving as a supervision signal to provide 3D geometric cues to our model. We also use the diffusion model as a powerful 2D multi-view feature extractor and construct 3D feature volumes from its intermediate representations. This transforms implicit 3D priors learned by the diffusion model into explicit 3D features. Beyond accurate keypoints estimation, we further introduce a pipeline that enables manipulation of 3D objects generated by the diffusion model. Experimental results on diverse aspects and datasets, including Human3.6M, Stanford Dogs, and several in-the-wild and out-of-domain datasets, highlight the effectiveness of our method in terms of accuracy, generalization, and its ability to enable manipulation of 3D objects generated by the diffusion model from a single image.