Low Bandwidth Video-Chat Compression using Deep Generative Models

TL;DR

This paper introduces a deep generative model-based method for low bandwidth video chat that reconstructs faces on the receiver's device using transmitted facial landmarks, enabling real-time video calls at significantly reduced data rates.

Contribution

It presents a novel, mobile-compatible deep generative architecture using facial landmarks for face reconstruction in video chat, achieving ultra-low bandwidth requirements.

Findings

Models compress to about 3MB, enabling real-time performance on smartphones.

Achieves video calling at a few kbits per second, much lower than existing methods.

Demonstrates trade-offs between quality and bandwidth using different landmark-based approaches.

Abstract

To unlock video chat for hundreds of millions of people hindered by poor connectivity or unaffordable data costs, we propose to authentically reconstruct faces on the receiver's device using facial landmarks extracted at the sender's side and transmitted over the network. In this context, we discuss and evaluate the benefits and disadvantages of several deep adversarial approaches. In particular, we explore quality and bandwidth trade-offs for approaches based on static landmarks, dynamic landmarks or segmentation maps. We design a mobile-compatible architecture based on the first order animation model of Siarohin et al. In addition, we leverage SPADE blocks to refine results in important areas such as the eyes and lips. We compress the networks down to about 3MB, allowing models to run in real time on iPhone 8 (CPU). This approach enables video calling at a few kbits per second, an order of magnitude lower than currently available alternatives.