Diffusion-based 3D Hand Motion Recovery with Intuitive Physics

TL;DR

This paper introduces a diffusion-based framework for 3D hand motion recovery from videos that incorporates physics knowledge, improving accuracy and temporal coherence without needing annotated video data.

Contribution

The novel physics-augmented diffusion model enhances 3D hand motion sequences using only motion capture data, improving upon existing image-based methods.

Findings

Achieves state-of-the-art performance on benchmarks.

Significantly improves temporal coherence in hand motion sequences.

Effectively integrates physics constraints into the diffusion process.

Abstract

While 3D hand reconstruction from monocular images has made significant progress, generating accurate and temporally coherent motion estimates from videos remains challenging, particularly during hand-object interactions. In this paper, we present a novel 3D hand motion recovery framework that enhances image-based reconstructions through a diffusion-based and physics-augmented motion refinement model. Our model captures the distribution of refined motion estimates conditioned on initial ones, generating improved sequences through an iterative denoising process. Instead of relying on scarce annotated video data, we train our model only using motion capture data without images. We identify valuable intuitive physics knowledge during hand-object interactions, including key motion states and their associated motion constraints. We effectively integrate these physical insights into our diffusion model to improve its performance. Extensive experiments demonstrate that our approach significantly improves various frame-wise reconstruction methods, achieving state-of-the-art (SOTA) performance on existing benchmarks.