SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers

TL;DR

SkyReels-Audio is a novel framework that synthesizes high-quality, temporally coherent talking portrait videos conditioned on multimodal inputs, enabling flexible editing and long-duration generation with improved lip-sync and facial consistency.

Contribution

It introduces a unified multimodal diffusion-based framework with a hybrid curriculum learning strategy and new loss functions for high-fidelity, controllable, and long-duration talking portrait video synthesis.

Findings

Achieves superior lip-sync accuracy and identity consistency

Ensures visual fidelity and temporal coherence in extended videos

Performs well under complex and challenging conditions

Abstract

The generation and editing of audio-conditioned talking portraits guided by multimodal inputs, including text, images, and videos, remains under explored. In this paper, we present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos. Built upon pretrained video diffusion transformers, our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs. We employ a hybrid curriculum learning strategy to progressively align audio with facial motion, enabling fine-grained multimodal control over long video sequences. To enhance local facial coherence, we introduce a facial mask loss and an audio-guided classifier-free guidance mechanism. A sliding-window denoising approach further fuses latent representations across temporal segments, ensuring visual fidelity and temporal consistency across extended durations and diverse identities. More importantly, we construct a dedicated data pipeline for curating high-quality triplets consisting of synchronized audio, video, and textual descriptions. Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior performance in lip-sync accuracy, identity consistency, and realistic facial dynamics, particularly under complex and challenging conditions.