OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation

TL;DR

OmniShow is an end-to-end framework that synthesizes high-quality human-object interaction videos conditioned on multiple modalities, advancing the state-of-the-art in multimodal video generation.

Contribution

The paper introduces OmniShow, a novel multimodal video generation framework with new conditioning techniques and a multi-stage training strategy, plus a comprehensive HOIVG benchmark.

Findings

OmniShow achieves state-of-the-art performance across various multimodal conditions.

The framework effectively harmonizes text, images, audio, and pose for video synthesis.

The HOIVG-Bench provides a new standard for evaluating human-object interaction video generation.

Abstract

In this work, we study Human-Object Interaction Video Generation (HOIVG), which aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose. This task holds significant practical value for automating content creation in real-world applications, such as e-commerce demonstrations, short video production, and interactive entertainment. However, existing approaches fail to accommodate all these requisite conditions. We present OmniShow, an end-to-end framework tailored for this practical yet challenging task, capable of harmonizing multimodal conditions and delivering industry-grade performance. To overcome the trade-off between controllability and quality, we introduce Unified Channel-wise Conditioning for efficient image and pose injection, and Gated Local-Context Attention to ensure precise audio-visual synchronization. To effectively address data scarcity, we develop a Decoupled-Then-Joint Training strategy that leverages a multi-stage training process with model merging to efficiently harness heterogeneous sub-task datasets. Furthermore, to fill the evaluation gap in this field, we establish HOIVG-Bench, a dedicated and comprehensive benchmark for HOIVG. Extensive experiments demonstrate that OmniShow achieves overall state-of-the-art performance across various multimodal conditioning settings, setting a solid standard for the emerging HOIVG task.