StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator

TL;DR

StyleSync is a novel framework that achieves high-fidelity, personalized lip synchronization in style-based generators, effectively balancing quality and generalization for both one-shot and few-shot scenarios.

Contribution

It introduces a style-based generator with a mask-guided encoding and style space refinement, enabling personalized, high-quality lip sync with limited data.

Findings

Outperforms existing methods in lip sync quality

Preserves identity and talking style accurately

Effective in diverse scenes and data scenarios

Abstract

Despite recent advances in syncing lip movements with any audio waves, current methods still struggle to balance generation quality and the model's generalization ability. Previous studies either require long-term data for training or produce a similar movement pattern on all subjects with low quality. In this paper, we propose StyleSync, an effective framework that enables high-fidelity lip synchronization. We identify that a style-based generator would sufficiently enable such a charming property on both one-shot and few-shot scenarios. Specifically, we design a mask-guided spatial information encoding module that preserves the details of the given face. The mouth shapes are accurately modified by audio through modulated convolutions. Moreover, our design also enables personalized lip-sync by introducing style space and generator refinement on only limited frames. Thus the identity and talking style of a target person could be accurately preserved. Extensive experiments demonstrate the effectiveness of our method in producing high-fidelity results on a variety of scenes. Resources can be found at https://hangz-nju-cuhk.github.io/projects/StyleSync.