End-to-End Learning-based Video Streaming Enhancement Pipeline: A Generative AI Approach

TL;DR

ELVIS is an end-to-end learning-based video streaming enhancement pipeline that leverages generative AI for improved quality and efficiency, balancing high video quality with smooth playback.

Contribution

It introduces a modular architecture combining server-side encoding and client-side generative in-painting, enabling adaptable and improved video streaming performance.

Findings

Up to 11 VMAF points improvement over baseline

Modular design allows integration of different codecs and models

Challenges remain for real-time deployment due to computational demands

Abstract

The primary challenge of video streaming is to balance high video quality with smooth playback. Traditional codecs are well tuned for this trade-off, yet their inability to use context means they must encode the entire video data and transmit it to the client. This paper introduces ELVIS (End-to-end Learning-based VIdeo Streaming Enhancement Pipeline), an end-to-end architecture that combines server-side encoding optimizations with client-side generative in-painting to remove and reconstruct redundant video data. Its modular design allows ELVIS to integrate different codecs, inpainting models, and quality metrics, making it adaptable to future innovations. Our results show that current technologies achieve improvements of up to 11 VMAF points over baseline benchmarks, though challenges remain for real-time applications due to computational demands. ELVIS represents a foundational step toward incorporating generative AI into video streaming pipelines, enabling higher quality experiences without increased bandwidth requirements.