iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformer

TL;DR

This paper introduces iDiT-HOI, a novel inpainting-based video diffusion transformer framework for realistic hand-object interaction reenactment, capable of generalizing to unseen objects and supporting long video generation.

Contribution

The paper proposes a unified inpainting token process and a two-stage diffusion transformer that enhances realism and generalization in HOI reenactment without extra parameters.

Findings

Outperforms existing methods in real-world scenes

Enables strong generalization to unseen objects and scenarios

Supports long-duration video generation

Abstract

Digital human video generation is gaining traction in fields like education and e-commerce, driven by advancements in head-body animation and lip-syncing technologies. However, realistic Hand-Object Interaction (HOI) - the complex dynamics between human hands and objects - continues to pose challenges. Generating natural and believable HOI reenactments is difficult due to issues such as occlusion between hands and objects, variations in object shapes and orientations, and the necessity for precise physical interactions, and importantly, the ability to generalize to unseen humans and objects. This paper presents a novel framework iDiT-HOI that enables in-the-wild HOI reenactment generation. Specifically, we propose a unified inpainting-based token process method, called Inp-TPU, with a two-stage video diffusion transformer (DiT) model. The first stage generates a key frame by inserting the designated object into the hand region, providing a reference for subsequent frames. The second stage ensures temporal coherence and fluidity in hand-object interactions. The key contribution of our method is to reuse the pretrained model's context perception capabilities without introducing additional parameters, enabling strong generalization to unseen objects and scenarios, and our proposed paradigm naturally supports long video generation. Comprehensive evaluations demonstrate that our approach outperforms existing methods, particularly in challenging real-world scenes, offering enhanced realism and more seamless hand-object interactions.