UniSync: Towards Generalizable and High-Fidelity Lip Synchronization for Challenging Scenarios

TL;DR

UniSync is a novel lip synchronization framework that combines mask-free and mask-based techniques, achieving high-fidelity, adaptable, and realistic talking videos across diverse real-world scenarios and challenging conditions.

Contribution

The paper introduces UniSync, a unified approach that integrates pose-anchored training and blending inference, along with domain adaptation and a new benchmark for real-world evaluation.

Findings

Outperforms state-of-the-art methods in diverse scenarios

Handles complex conditions like stylized avatars and occlusions

Demonstrates high domain adaptability and realism

Abstract

Lip synchronization aims to generate realistic talking videos that match given audio, which is essential for high-quality video dubbing. However, current methods have fundamental drawbacks: mask-based approaches suffer from local color discrepancies, while mask-free methods struggle with global background texture misalignment. Furthermore, most methods struggle with diverse real-world scenarios such as stylized avatars, face occlusion, and extreme lighting conditions. In this paper, we propose UniSync, a unified framework designed for achieving high-fidelity lip synchronization in diverse scenarios. Specifically, UniSync uses a mask-free pose-anchored training strategy to keep head motion and eliminate synthesis color artifacts, while employing mask-based blending consistent inference to ensure structural precision and smooth blending. Notably, fine-tuning on compact but diverse videos empowers our model with exceptional domain adaptability, handling complex corner cases effectively. We also introduce the RealWorld-LipSync benchmark to evaluate models under real-world demands, which covers diverse application scenarios including both human faces and stylized avatars. Extensive experiments demonstrate that UniSync significantly outperforms state-of-the-art methods, advancing the field towards truly generalizable and production-ready lip synchronization.