LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision

TL;DR

This paper improves audio-driven lip-sync in portrait animation by integrating SyncNet supervision into latent diffusion models, addressing shortcut learning, and introducing StableSyncNet and TREPA for better convergence and temporal consistency, achieving state-of-the-art results.

Contribution

It presents a novel approach to enhance lip-sync accuracy by explicitly enforcing audio-visual correlation through SyncNet supervision and introduces StableSyncNet and TREPA mechanisms.

Findings

StableSyncNet improves SyncNet convergence from 91% to 94%.

The proposed method surpasses state-of-the-art lip-sync methods on HDTF and VoxCeleb2 datasets.

Enhanced temporal consistency in generated videos with TREPA.

Abstract

End-to-end audio-conditioned latent diffusion models (LDMs) have been widely adopted for audio-driven portrait animation, demonstrating their effectiveness in generating lifelike and high-resolution talking videos. However, direct application of audio-conditioned LDMs to lip-synchronization (lip-sync) tasks results in suboptimal lip-sync accuracy. Through an in-depth analysis, we identified the underlying cause as the "shortcut learning problem", wherein the model predominantly learns visual-visual shortcuts while neglecting the critical audio-visual correlations. To address this issue, we explored different approaches for integrating SyncNet supervision into audio-conditioned LDMs to explicitly enforce the learning of audio-visual correlations. Since the performance of SyncNet directly influences the lip-sync accuracy of the supervised model, the training of a well-converged SyncNet becomes crucial. We conducted the first comprehensive empirical studies to identify key factors affecting SyncNet convergence. Based on our analysis, we introduce StableSyncNet, with an architecture designed for stable convergence. Our StableSyncNet achieved a significant improvement in accuracy, increasing from 91% to 94% on the HDTF test set. Additionally, we introduce a novel Temporal Representation Alignment (TREPA) mechanism to enhance temporal consistency in the generated videos. Experimental results show that our method surpasses state-of-the-art lip-sync approaches across various evaluation metrics on the HDTF and VoxCeleb2 datasets.