Audio-Visual Driven Compression for Low-Bitrate Talking Head Videos

TL;DR

This paper introduces an audio-visual driven codec that enhances low-bitrate talking head video compression by integrating 3D motion features and audio signals, leading to better quality and lip-sync at reduced bitrates.

Contribution

It presents a novel codec combining audio and visual cues with 3D motion features to improve low-bitrate talking head video compression, outperforming existing methods.

Findings

Reduces bitrate by 22% compared to VVC.

Achieves 8.5% bitrate reduction over state-of-the-art learning-based codecs.

Provides superior lip-sync and visual fidelity at similar bitrates.

Abstract

Talking head video compression has advanced with neural rendering and keypoint-based methods, but challenges remain, especially at low bit rates, including handling large head movements, suboptimal lip synchronization, and distorted facial reconstructions. To address these problems, we propose a novel audio-visual driven video codec that integrates compact 3D motion features and audio signals. This approach robustly models significant head rotations and aligns lip movements with speech, improving both compression efficiency and reconstruction quality. Experiments on the CelebV-HQ dataset show that our method reduces bitrate by 22% compared to VVC and by 8.5% over state-of-the-art learning-based codec. Furthermore, it provides superior lip-sync accuracy and visual fidelity at comparable bitrates, highlighting its effectiveness in bandwidth-constrained scenarios.