Real-time One-Step Diffusion-based Expressive Portrait Videos Generation

TL;DR

This paper introduces OSA-LCM, a real-time diffusion-based model for expressive portrait videos that generates high-quality videos in a single step, significantly improving speed while maintaining quality.

Contribution

The paper proposes a novel one-step diffusion model with a specialized discriminator and a second-stage training method to enable real-time, high-quality portrait video generation.

Findings

Achieves comparable quality to multi-step models in a single step

Over 10x faster than existing diffusion-based methods

Outperforms current open-source portrait video models

Abstract

Latent diffusion models have made great strides in generating expressive portrait videos with accurate lip-sync and natural motion from a single reference image and audio input. However, these models are far from real-time, often requiring many sampling steps that take minutes to generate even one second of video-significantly limiting practical use. We introduce OSA-LCM (One-Step Avatar Latent Consistency Model), paving the way for real-time diffusion-based avatars. Our method achieves comparable video quality to existing methods but requires only one sampling step, making it more than 10x faster. To accomplish this, we propose a novel avatar discriminator design that guides lip-audio consistency and motion expressiveness to enhance video quality in limited sampling steps. Additionally, we employ a second-stage training architecture using an editing fine-tuned method (EFT), transforming video generation into an editing task during training to effectively address the temporal gap challenge in single-step generation. Experiments demonstrate that OSA-LCM outperforms existing open-source portrait video generation models while operating more efficiently with a single sampling step.