A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generation

TL;DR

This paper introduces a lightweight, efficient speech-driven talking-face generation model that reduces computational costs significantly while maintaining high-quality output through knowledge distillation and mixed precision quantization.

Contribution

The study proposes a compact generator architecture with a novel training scheme and mixed precision quantization to enhance efficiency without sacrificing quality.

Findings

28× reduction in parameters and MACs

Up to 19× speedup on edge GPUs

Performance comparable to original models

Abstract

Virtual humans have gained considerable attention in numerous industries, e.g., entertainment and e-commerce. As a core technology, synthesizing photorealistic face frames from target speech and facial identity has been actively studied with generative adversarial networks. Despite remarkable results of modern talking-face generation models, they often entail high computational burdens, which limit their efficient deployment. This study aims to develop a lightweight model for speech-driven talking-face synthesis. We build a compact generator by removing the residual blocks and reducing the channel width from Wav2Lip, a popular talking-face generator. We also present a knowledge distillation scheme to stably yet effectively train the small-capacity generator without adversarial learning. We reduce the number of parameters and MACs by 28$\times$ while retaining the performance of the original model. Moreover, to alleviate a severe performance drop when converting the whole generator to INT8 precision, we adopt a selective quantization method that uses FP16 for the quantization-sensitive layers and INT8 for the other layers. Using this mixed precision, we achieve up to a 19$\times$ speedup on edge GPUs without noticeably compromising the generation quality.