Audio-Visual Cross-Modal Compression for Generative Face Video Coding

TL;DR

This paper introduces AVCC, a novel framework that jointly compresses audio and video for face videos, leveraging cross-modal coherence to improve rate-distortion performance in low bitrate scenarios.

Contribution

The paper presents a unified audio-visual compression method that exploits audio-visual correlation via a diffusion process, enabling synchronized reconstruction and cross-modal generation.

Findings

AVCC outperforms VVC and existing GFVC methods in rate-distortion metrics.

In low-rate scenarios, AVCC can reconstruct one modality from the other.

The framework effectively exploits audio-visual coherence for efficient compression.

Abstract

Generative face video coding (GFVC) is vital for modern applications like video conferencing, yet existing methods primarily focus on video motion while neglecting the significant bitrate contribution of audio. Despite the well-established correlation between audio and lip movements, this cross-modal coherence has not been systematically exploited for compression. To address this, we propose an Audio-Visual Cross-Modal Compression (AVCC) framework that jointly compresses audio and video streams. Our framework extracts motion information from video and tokenizes audio features, then aligns them through a unified audio-video diffusion process. This allows synchronized reconstruction of both modalities from a shared representation. In extremely low-rate scenarios, AVCC can even reconstruct one modality from the other. Experiments show that AVCC significantly outperforms the Versatile Video Coding (VVC) standard and state-of-the-art GFVC schemes in rate-distortion performance, paving the way for more efficient multimodal communication systems.