HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching

TL;DR

HO-Flow is a novel framework that synthesizes realistic, temporally coherent 3D hand-object interactions from text and 3D object data, advancing motion generation in vision and robotics.

Contribution

It introduces a unified latent representation and a flow matching model for improved generalization and temporal reasoning in hand-object interaction synthesis.

Findings

Achieves state-of-the-art results on GRAB, OakInk, and DexYCB benchmarks.

Effectively models rich interaction dynamics and motion diversity.

Enhances generalization through relative object motion prediction.

Abstract

Generating realistic 3D hand-object interactions (HOI) is a fundamental challenge in computer vision and robotics, requiring both temporal coherence and high-fidelity physical plausibility. Existing methods remain limited in their ability to learn expressive motion representations for generation and perform temporal reasoning. In this paper, we present HO-Flow, a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects. HO-Flow first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold by incorporating hand and object kinematics, enabling the representation to capture rich interaction dynamics. It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation, improving temporal coherence. To further enhance generalization, HO-Flow predicts object motions relative to the initial frame, enabling effective pre-training on large-scale synthetic data. Experiments on the GRAB, OakInk, and DexYCB benchmarks demonstrate that HO-Flow achieves state-of-the-art performance in both physical plausibility and motion diversity for interaction motion synthesis.