Learning in situ: a randomized experiment in video streaming

TL;DR

This study conducts a large-scale randomized trial of video streaming algorithms, revealing challenges for machine learning approaches in real-world conditions and proposing a hybrid control method that outperforms existing schemes.

Contribution

It provides empirical evidence on the performance of streaming algorithms in real-world settings and introduces a robust hybrid ABR algorithm combining classical control with learned network prediction.

Findings

Simple buffer-based control often outperforms complex algorithms in practice.

Heavy-tailed network behavior complicates the deployment of learned algorithms.

The proposed hybrid algorithm outperforms other schemes in real-world tests.

Abstract

We describe the results of a randomized controlled trial of video-streaming algorithms for bitrate selection and network prediction. Over the last eight months, we have streamed 14.2 years of video to 56,000 users across the Internet. Sessions are randomized in blinded fashion among algorithms, and client telemetry is recorded for analysis. We found that in this real-world setting, it is difficult for sophisticated or machine-learned control schemes to outperform a "simple" scheme (buffer-based control), notwithstanding good performance in network emulators or simulators. We performed a statistical analysis and found that the variability and heavy-tailed nature of network and algorithm behavior create hurdles for robust learned algorithms in this area. We developed an ABR algorithm that robustly outperforms other schemes in practice, by combining classical control with a learned network predictor, trained with supervised learning in situ on data from the real deployment environment. To support further investigation, we are publishing an archive of traces and results each day, and will open our ongoing study to the community. We welcome other researchers to use this platform to develop and validate new algorithms for bitrate selection, network prediction, and congestion control.