FAIVConf: Face enhancement for AI-based Video Conference with Low Bit-rate

TL;DR

FAIVConf is a neural face generation-based video conferencing framework that significantly reduces bit-rate and enhances visual quality by employing face-swapping, facial blurring, and dynamic source updates.

Contribution

It introduces a novel neural face enhancement framework with techniques like face-swapping and dynamic updates to improve robustness and efficiency in low bit-rate video conferencing.

Findings

Achieves significant bit-rate reduction compared to H.264 and H.265.

Provides better visual quality at the same bit-rate.

Enhances robustness in real-world video conference scenarios.

Abstract

Recently, high-quality video conferencing with fewer transmission bits has become a very hot and challenging problem. We propose FAIVConf, a specially designed video compression framework for video conferencing, based on the effective neural human face generation techniques. FAIVConf brings together several designs to improve the system robustness in real video conference scenarios: face-swapping to avoid artifacts in background animation; facial blurring to decrease transmission bit-rate and maintain the quality of extracted facial landmarks; and dynamic source update for face view interpolation to accommodate a large range of head poses. Our method achieves a significant bit-rate reduction in the video conference and gives much better visual quality under the same bit-rate compared with H.264 and H.265 coding schemes.