JoyStreamer: Unlocking Highly Expressive Avatars via Harmonized Text-Audio Conditioning

TL;DR

JoyStreamer is a novel framework that significantly enhances avatar video generation by improving text and audio alignment, enabling complex, natural, and coherent full-body avatar motions with dynamic camera control.

Contribution

The paper introduces a twin-teacher training algorithm and dynamic multi-modal conditioning modulation to improve avatar generation capabilities.

Findings

Outperforms state-of-the-art models like Omnihuman-1.5 and KlingAvatar 2.0.

Enables complex multi-person dialogues and non-human role-playing.

Produces natural, temporally coherent full-body motions with dynamic camera movements.

Abstract

Existing video avatar models have demonstrated impressive capabilities in scenarios such as talking, public speaking, and singing. However, the majority of these methods exhibit limited alignment with respect to text instructions, particularly when the prompts involve complex elements including large full-body movement, dynamic camera trajectory, background transitions, or human-object interactions. To break out this limitation, we present JoyAvatar, a framework capable of generating long duration avatar videos, featuring two key technical innovations. Firstly, we introduce a twin-teacher enhanced training algorithm that enables the model to transfer inherent text-controllability from the foundation model while simultaneously learning audio-visual synchronization. Secondly, during training, we dynamically modulate the strength of multi-modal conditions (e.g., audio and text) based on the distinct denoising timestep, aiming to mitigate conflicts between the heterogeneous conditioning signals. These two key designs serve to substantially expand the avatar model's capacity to generate natural, temporally coherent full-body motions and dynamic camera movements as well as preserve the basic avatar capabilities, such as accurate lip-sync and identity consistency. GSB evaluation results demonstrate that our JoyStreamer model outperforms the state-of-the-art models such as Omnihuman-1.5 and KlingAvatar 2.0. Moreover, our approach enables complex applications including multi-person dialogues and non-human subjects role-playing. Some video samples are provided on https://joystreamer.github.io/.