HyperLips: Hyper Control Lips with High Resolution Decoder for Talking Face Generation

TL;DR

HyperLips is a two-stage framework that enhances talking face generation by controlling lip movements with a hypernetwork and producing high-resolution, realistic videos with a dedicated decoder, outperforming existing methods.

Contribution

The paper introduces HyperLips, a novel two-stage approach combining a hypernetwork for lip control and a high-resolution decoder for improved visual quality in talking face generation.

Findings

Outperforms state-of-the-art methods in realism and lip synchronization.

Produces high-fidelity facial videos with better visual quality.

Demonstrates effectiveness through extensive experiments.

Abstract

Talking face generation has a wide range of potential applications in the field of virtual digital humans. However, rendering high-fidelity facial video while ensuring lip synchronization is still a challenge for existing audio-driven talking face generation approaches. To address this issue, we propose HyperLips, a two-stage framework consisting of a hypernetwork for controlling lips and a high-resolution decoder for rendering high-fidelity faces. In the first stage, we construct a base face generation network that uses the hypernetwork to control the encoding latent code of the visual face information over audio. First, FaceEncoder is used to obtain latent code by extracting features from the visual face information taken from the video source containing the face frame.Then, HyperConv, which weighting parameters are updated by HyperNet with the audio features as input, will modify the latent code to synchronize the lip movement with the audio. Finally, FaceDecoder will decode the modified and synchronized latent code into visual face content. In the second stage, we obtain higher quality face videos through a high-resolution decoder. To further improve the quality of face generation, we trained a high-resolution decoder, HRDecoder, using face images and detected sketches generated from the first stage as input.Extensive quantitative and qualitative experiments show that our method outperforms state-of-the-art work with more realistic, high-fidelity, and lip synchronization. Project page: https://semchan.github.io/HyperLips Project/