Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips

TL;DR

This paper presents a method for reconstructing 3D hand-object interactions from short videos by combining per-video optimization with a diffusion-based prior, improving accuracy over previous methods.

Contribution

It introduces a diffusion network as a prior for 3D reconstruction of hand-object interactions, enhancing multi-view inference with learned geometric priors.

Findings

Significant improvements over prior methods in 3D reconstruction accuracy.

Effective reconstruction of arbitrary YouTube clips of hand-object interactions.

Robust performance across six object categories in egocentric videos.

Abstract

We tackle the task of reconstructing hand-object interactions from short video clips. Given an input video, our approach casts 3D inference as a per-video optimization and recovers a neural 3D representation of the object shape, as well as the time-varying motion and hand articulation. While the input video naturally provides some multi-view cues to guide 3D inference, these are insufficient on their own due to occlusions and limited viewpoint variations. To obtain accurate 3D, we augment the multi-view signals with generic data-driven priors to guide reconstruction. Specifically, we learn a diffusion network to model the conditional distribution of (geometric) renderings of objects conditioned on hand configuration and category label, and leverage it as a prior to guide the novel-view renderings of the reconstructed scene. We empirically evaluate our approach on egocentric videos across 6 object categories, and observe significant improvements over prior single-view and multi-view methods. Finally, we demonstrate our system's ability to reconstruct arbitrary clips from YouTube, showing both 1st and 3rd person interactions.