Admission Control for A Single Server Waiting Time Process in Heavy Traffic

TL;DR

This paper studies an optimal control problem for a single-server queue in heavy traffic, deriving asymptotically optimal policies using diffusion approximations and exploring reinforcement learning methods for solution approximation.

Contribution

It introduces a diffusion control framework for queue management in heavy traffic and connects theoretical optimal strategies with RL algorithms like REINFORCE.

Findings

Derived the heavy traffic limit as a diffusion control problem.

Established asymptotically optimal control policies.

Compared theoretical and data-driven solution approaches.

Abstract

We address a single server queue control problem (QCP) in heavy traffic originating from Lee and Weerasinghe (2011). The state process represents the offered waiting time, the customer arrival has a state-dependent intensity, and the customers' service and patience times are i.i.d with general distributions. We introduce an infinite-horizon discounted cost functional consisting of a control cost generated from the use of control and a penalty for idleness cost. Our primary goal is to tackle the QCP, taking into account a non-trivial control cost and a non-increasing cost function resulting from the control mechanisms in the waiting time. Under mild assumptions, the heavy traffic limit of the QCP yields a stochastic control problem described by a diffusion process, which we call a diffusion control problem (DCP). We find the optimal control of the associated DCP by incorporating the Legendre-Fenchel transform and a formal Hamilton-Jacobi-Bellman (HJB) equation. Then, we ``translate'' this optimal strategy to the QCP, of which we obtain an asymptotically optimal policy. Apart from theoretical results, we also examine the REINFORCE algorithm, a Reinforcement learning (RL) approach, for solving stochastic controls motivated by recent literature. We highlight the advantages and limitations of simulation from theoretical results and data-driven algorithms.