DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformers

TL;DR

DreamActor-H1 introduces a diffusion transformer framework for generating high-fidelity human-product demonstration videos that accurately preserve identities, details, and spatial relationships, enhancing realism and consistency.

Contribution

The paper presents a novel diffusion transformer model that integrates paired reference information, motion guidance, and structured text encoding for realistic human-product video synthesis.

Findings

Outperforms state-of-the-art methods in identity preservation.

Generates realistic and consistent demonstration motions.

Effectively models human-product spatial relationships.

Abstract

In e-commerce and digital marketing, generating high-fidelity human-product demonstration videos is important for effective product presentation. However, most existing frameworks either fail to preserve the identities of both humans and products or lack an understanding of human-product spatial relationships, leading to unrealistic representations and unnatural interactions. To address these challenges, we propose a Diffusion Transformer (DiT)-based framework. Our method simultaneously preserves human identities and product-specific details, such as logos and textures, by injecting paired human-product reference information and utilizing an additional masked cross-attention mechanism. We employ a 3D body mesh template and product bounding boxes to provide precise motion guidance, enabling intuitive alignment of hand gestures with product placements. Additionally, structured text encoding is used to incorporate category-level semantics, enhancing 3D consistency during small rotational changes across frames. Trained on a hybrid dataset with extensive data augmentation strategies, our approach outperforms state-of-the-art techniques in maintaining the identity integrity of both humans and products and generating realistic demonstration motions. Project page: https://lizhenwangt.github.io/DreamActor-H1/.