Teacher-Student Diffusion Model for Text-Driven 3D Hand Motion Generation

TL;DR

This paper introduces TSHaMo, a diffusion-based framework that enables realistic, diverse 3D hand motion generation from text by leveraging a teacher-student training strategy with auxiliary signals, without needing 3D object meshes at inference.

Contribution

The paper presents a novel teacher-student diffusion model for text-driven 3D hand motion generation that improves quality and diversity without requiring 3D objects during testing.

Findings

TSHaMo outperforms existing methods on GRAB and H2O datasets.

The model maintains robustness with diverse auxiliary inputs.

It effectively generates detailed hand gestures from natural language.

Abstract

Generating realistic 3D hand motion from natural language is vital for VR, robotics, and human-computer interaction. Existing methods either focus on full-body motion, overlooking detailed hand gestures, or require explicit 3D object meshes, limiting generality. We propose TSHaMo, a model-agnostic teacher-student diffusion framework for text-driven hand motion generation. The student model learns to synthesize motions from text alone, while the teacher leverages auxiliary signals (e.g., MANO parameters) to provide structured guidance during training. A co-training strategy enables the student to benefit from the teacher's intermediate predictions while remaining text-only at inference. Evaluated using two diffusion backbones on GRAB and H2O, TSHaMo consistently improves motion quality and diversity. Ablations confirm its robustness and flexibility in using diverse auxiliary inputs without requiring 3D objects at test time.