Better than the Best: Gradient-based Improper Reinforcement Learning for Network Scheduling

TL;DR

This paper introduces a gradient-based reinforcement learning approach for network scheduling that outperforms existing policies, adapts to nonstationary traffic, and stabilizes unstable systems, addressing the complexity of modern communication networks.

Contribution

It proposes a novel top-down reinforcement learning method for network scheduling that surpasses traditional policies and provides theoretical convergence and finite-time performance guarantees.

Findings

Algorithm outperforms existing policies in simulations.

Effective in nonstationary traffic conditions.

Can stabilize systems with unstable policies.

Abstract

We consider the problem of scheduling in constrained queueing networks with a view to minimizing packet delay. Modern communication systems are becoming increasingly complex, and are required to handle multiple types of traffic with widely varying characteristics such as arrival rates and service times. This, coupled with the need for rapid network deployment, render a bottom up approach of first characterizing the traffic and then devising an appropriate scheduling protocol infeasible. In contrast, we formulate a top down approach to scheduling where, given an unknown network and a set of scheduling policies, we use a policy gradient based reinforcement learning algorithm that produces a scheduler that performs better than the available atomic policies. We derive convergence results and analyze finite time performance of the algorithm. Simulation results show that the algorithm performs well even when the arrival rates are nonstationary and can stabilize the system even when the constituent policies are unstable.