Lightweight High-Fidelity Low-Bitrate Talking Face Compression for 3D Video Conference

TL;DR

This paper presents a lightweight, high-fidelity 3D talking face compression method that combines parametric modeling and neural rendering to enable real-time, low-bitrate 3D video conferencing with high-quality facial details.

Contribution

It introduces a novel compression framework integrating FLAME-based modeling with neural rendering and a compact representation scheme for efficient real-time transmission.

Findings

Achieves superior rate-distortion performance at low bitrates.

Enables real-time high-quality 3D facial rendering.

Reduces computational costs compared to implicit neural rendering methods.

Abstract

The demand for immersive and interactive communication has driven advancements in 3D video conferencing, yet achieving high-fidelity 3D talking face representation at low bitrates remains a challenge. Traditional 2D video compression techniques fail to preserve fine-grained geometric and appearance details, while implicit neural rendering methods like NeRF suffer from prohibitive computational costs. To address these challenges, we propose a lightweight, high-fidelity, low-bitrate 3D talking face compression framework that integrates FLAME-based parametric modeling with 3DGS neural rendering. Our approach transmits only essential facial metadata in real time, enabling efficient reconstruction with a Gaussian-based head model. Additionally, we introduce a compact representation and compression scheme, including Gaussian attribute compression and MLP optimization, to enhance transmission efficiency. Experimental results demonstrate that our method achieves superior rate-distortion performance, delivering high-quality facial rendering at extremely low bitrates, making it well-suited for real-time 3D video conferencing applications.