Learning and Information in Stochastic Networks and Queues

TL;DR

This paper reviews how information and learning techniques like reinforcement learning and bandits influence the stability and optimization of queueing systems, connecting queueing policies with learning theory and discussing recent advances and challenges.

Contribution

It establishes a theoretical link between queueing policies and Blackwell's Approachability Theorem, and discusses the integration of statistical learning and reinforcement learning in queueing systems.

Findings

MaxWeight and BackPressure policies relate to Blackwell's Approachability Theorem.

Queue size regret can be bounded using perceptron-based service classification.

Differentiates roles of epistemic and aleatoric information in decision making.

Abstract

We review the role of information and learning in the stability and optimization of queueing systems. In recent years, techniques from supervised learning, bandit learning and reinforcement learning have been applied to queueing systems supported by increasing role of information in decision making. We present observations and new results that help rationalize the application of these areas to queueing systems. We prove that the MaxWeight and BackPressure policies are an application of Blackwell's Approachability Theorem. This connects queueing theoretic results with adversarial learning. We then discuss the requirements of statistical learning for service parameter estimation. As an example, we show how queue size regret can be bounded when applying a perceptron algorithm to classify service. Next, we discuss the role of state information in improved decision making. Here we contrast the roles of epistemic information (information on uncertain parameters) and aleatoric information (information on an uncertain state). Finally we review recent advances in the theory of reinforcement learning and queueing, as well as, provide discussion on current research challenges.