Offline to Online Learning for Real-Time Bandwidth Estimation

TL;DR

Merlin is an imitation learning-based approach that transforms heuristic bandwidth estimation algorithms into neural networks, enabling efficient offline training and effective online personalization for real-time video applications.

Contribution

The paper introduces Merlin, a novel imitation learning framework that converts heuristic policies into neural networks for data-driven offline training and online adaptation in bandwidth estimation.

Findings

Merlin matches heuristic policy QoE without significant difference.

Finetuning Merlin improves QoE by up to 7.8%.

IL-based approach converges with 80% fewer samples than RL methods.

Abstract

Real-time video applications require accurate bandwidth estimation (BWE) to maintain user experience across varying network conditions. However, increasing network heterogeneity challenges general-purpose BWE algorithms, necessitating solutions that adapt to end-user environments. While widely adopted, heuristic-based methods are difficult to individualize without extensive domain expertise. Conversely, online reinforcement learning (RL) offers ease of customization but neglects prior domain expertise and suffers from sample inefficiency. Thus, we present Merlin, an imitation learning-based solution that replaces the manual parameter tuning of heuristic-based methods with data-driven updates to streamline end-user personalization. Our key insight is that transforming heuristic-based BWE algorithms into neural networks facilitates data-driven personalization. Merlin utilizes Behavioral Cloning to efficiently learn from offline telemetry logs, capturing heuristic policies without live network interactions. The cloned policy can then be seamlessly tailored to end user network conditions through online finetuning. In real intercontinental videoconferencing calls, Merlin matches our heuristic's policy with no statistically significant differences in user quality of experience (QoE). Finetuning Merlin's control policy to end-user environments enables QoE improvements of up to 7.8% compared to the heuristic policy. Lastly, our IL-based design performs competitively with current state-of-the-art online RL techniques but converges with 80% fewer videoconferencing samples, facilitating practical end-user personalization.