Real-world Video Adaptation with Reinforcement Learning

TL;DR

This paper presents a reinforcement learning-based adaptive bitrate algorithm for client-side video streaming that outperforms traditional methods in real-world deployment, addressing practical challenges with customized neural network design and optimization techniques.

Contribution

The paper introduces a scalable neural network architecture and training method tailored for real-world video streaming, along with a reward shaping approach using Bayesian optimization.

Findings

RL-based ABR outperforms existing algorithms in large-scale deployment

Customized neural network supports videos with arbitrary bitrates

Effective reward shaping improves QoE optimization

Abstract

Client-side video players employ adaptive bitrate (ABR) algorithms to optimize user quality of experience (QoE). We evaluate recently proposed RL-based ABR methods in Facebook's web-based video streaming platform. Real-world ABR contains several challenges that requires customized designs beyond off-the-shelf RL algorithms -- we implement a scalable neural network architecture that supports videos with arbitrary bitrate encodings; we design a training method to cope with the variance resulting from the stochasticity in network conditions; and we leverage constrained Bayesian optimization for reward shaping in order to optimize the conflicting QoE objectives. In a week-long worldwide deployment with more than 30 million video streaming sessions, our RL approach outperforms the existing human-engineered ABR algorithms.