HANDI: Hand-Centric Text-and-Image Conditioned Video Generation

TL;DR

HANDI is a novel diffusion-based video generation method that emphasizes hand-centric actions, automatically identifying motion regions and refining hand poses to improve action clarity in complex environments.

Contribution

The paper introduces an automatic motion area generation method guided by visual context and text prompts, along with a Hand Refinement Loss for better hand pose consistency.

Findings

Significant improvement in hand motion clarity over state-of-the-art methods

Effective on challenging datasets like EpicKitchens and Ego4D

Demonstrates robustness across diverse environments and actions

Abstract

Despite the recent strides in video generation, state-of-the-art methods still struggle with elements of visual detail. One particularly challenging case is the class of videos in which the intricate motion of the hand coupled with a mostly stable and otherwise distracting environment is necessary to convey the execution of some complex action and its effects. To address these challenges, we introduce a new method for video generation that focuses on hand-centric actions. Our diffusion-based method incorporates two distinct innovations. First, we propose an automatic method to generate the motion area -- the region in the video in which the detailed activities occur -- guided by both the visual context and the action text prompt, rather than assuming this region can be provided manually as is now commonplace. Second, we introduce a critical Hand Refinement Loss to guide the diffusion model to focus on smooth and consistent hand poses. We evaluate our method on challenging augmented datasets based on EpicKitchens and Ego4D, demonstrating significant improvements over state-of-the-art methods in terms of action clarity, especially of the hand motion in the target region, across diverse environments and actions. Video results can be found in https://excitedbutter.github.io/project_page