Queue-Learning: A Reinforcement Learning Approach for Providing Quality of Service

TL;DR

This paper introduces a reinforcement learning-based service-rate control method that provides probabilistic delay guarantees in tandem queueing systems, improving QoS management without overusing resources.

Contribution

It presents a novel RL-based controller using DDPG that offers explicit probabilistic delay bounds, overcoming limitations of traditional queueing theory methods.

Findings

Successfully maintains delay bounds in non-exponential queueing systems.

Demonstrates effective resource utilization while meeting QoS constraints.

Validates approach through simulations of tandem queue systems.

Abstract

End-to-end delay is a critical attribute of quality of service (QoS) in application domains such as cloud computing and computer networks. This metric is particularly important in tandem service systems, where the end-to-end service is provided through a chain of services. Service-rate control is a common mechanism for providing QoS guarantees in service systems. In this paper, we introduce a reinforcement learning-based (RL-based) service-rate controller that provides probabilistic upper-bounds on the end-to-end delay of the system, while preventing the overuse of service resources. In order to have a general framework, we use queueing theory to model the service systems. However, we adopt an RL-based approach to avoid the limitations of queueing-theoretic methods. In particular, we use Deep Deterministic Policy Gradient (DDPG) to learn the service rates (action) as a function of the queue lengths (state) in tandem service systems. In contrast to existing RL-based methods that quantify their performance by the achieved overall reward, which could be hard to interpret or even misleading, our proposed controller provides explicit probabilistic guarantees on the end-to-end delay of the system. The evaluations are presented for a tandem queueing system with non-exponential inter-arrival and service times, the results of which validate our controller's capability in meeting QoS constraints.