Structured Reinforcement Learning for Media Streaming at the Wireless Edge

TL;DR

This paper develops a structured reinforcement learning approach for optimizing media streaming at the wireless edge, resulting in fast, low-complexity policies that significantly improve user experience.

Contribution

It introduces a threshold-based constrained reinforcement learning method for media streaming, leveraging problem structure for efficient policy learning and deployment.

Findings

Structured policy achieves over 30% QOE improvement.

Fast policy learning with low computational overhead (~15μs).

Converges to globally optimal policy using natural policy gradient.

Abstract

Media streaming is the dominant application over wireless edge (access) networks. The increasing softwarization of such networks has led to efforts at intelligent control, wherein application-specific actions may be dynamically taken to enhance the user experience. The goal of this work is to develop and demonstrate learning-based policies for optimal decision making to determine which clients to dynamically prioritize in a video streaming setting. We formulate the policy design question as a constrained Markov decision problem (CMDP), and observe that by using a Lagrangian relaxation we can decompose it into single-client problems. Further, the optimal policy takes a threshold form in the video buffer length, which enables us to design an efficient constrained reinforcement learning (CRL) algorithm to learn it. Specifically, we show that a natural policy gradient (NPG) based algorithm that is derived using the structure of our problem converges to the globally optimal policy. We then develop a simulation environment for training, and a real-world intelligent controller attached to a WiFi access point for evaluation. We empirically show that the structured learning approach enables fast learning. Furthermore, such a structured policy can be easily deployed due to low computational complexity, leading to policy execution taking only about 15$\mu$s. Using YouTube streaming experiments in a resource constrained scenario, we demonstrate that the CRL approach can increase quality of experience (QOE) by over 30\%.