StyleLipSync: Style-based Personalized Lip-sync Video Generation

TL;DR

StyleLipSync is a novel style-based model that generates personalized, high-quality lip-sync videos from arbitrary audio, leveraging a pre-trained StyleGAN and pose-aware masking for naturalness.

Contribution

It introduces a style-based generative approach with pose-aware masking and a few-shot adaptation method for personalized lip-sync video generation.

Findings

Accurately generates lip-sync videos in zero-shot settings.

Enhances unseen face characteristics with minimal target video.

Outperforms previous methods in naturalness and personalization.

Abstract

In this paper, we present StyleLipSync, a style-based personalized lip-sync video generative model that can generate identity-agnostic lip-synchronizing video from arbitrary audio. To generate a video of arbitrary identities, we leverage expressive lip prior from the semantically rich latent space of a pre-trained StyleGAN, where we can also design a video consistency with a linear transformation. In contrast to the previous lip-sync methods, we introduce pose-aware masking that dynamically locates the mask to improve the naturalness over frames by utilizing a 3D parametric mesh predictor frame by frame. Moreover, we propose a few-shot lip-sync adaptation method for an arbitrary person by introducing a sync regularizer that preserves lip-sync generalization while enhancing the person-specific visual information. Extensive experiments demonstrate that our model can generate accurate lip-sync videos even with the zero-shot setting and enhance characteristics of an unseen face using a few seconds of target video through the proposed adaptation method.