Multi-Agent Q-Learning Aided Backpressure Routing Algorithm for Delay Reduction

TL;DR

This paper introduces a multi-agent Q-learning aided backpressure routing algorithm that significantly reduces packet delay in queueing networks by relying only on local information, outperforming existing methods.

Contribution

It proposes a novel distributed routing algorithm using multi-agent Q-learning that improves delay performance without requiring global queue information.

Findings

Reduces average packet delay by 95% under light traffic.

Reduces average packet delay by 41% under moderate traffic.

Maintains throughput-optimality and low computational complexity.

Abstract

In queueing networks, it is well known that the throughput-optimal backpressure routing algorithm results in poor delay performance for light and moderate traffic loads. To improve delay performance, state-of-the-art backpressure routing algorithm (called BPmin [1]) exploits queue length information to direct packets to less congested routes to their destinations. However, BPmin algorithm estimates route congestion based on unrealistic assumption that every node in the network knows real-time global queue length information of all other nodes. In this paper, we propose multi-agent Q-learning aided backpressure routing algorithm, where each node estimates route congestion using only local information of neighboring nodes. Our algorithm not only outperforms state-of-the-art BPmin algorithm in delay performance but also retains the following appealing features: distributed implementation, low computation complexity and throughput-optimality. Simulation results show our algorithm reduces average packet delay by 95% for light traffic loads and by 41% for moderate traffic loads when compared to state-of-the-art BPmin algorithm.