Spatial-Temporal Reinforcement Learning for Network Routing with Non-Markovian Traffic

TL;DR

This paper introduces a spatial-temporal reinforcement learning framework for network routing that effectively handles non-Markovian traffic patterns and exploits network topology structure, outperforming traditional methods.

Contribution

The paper presents a novel STRL framework that models non-Markovian traffic and spatial network features, improving routing performance over existing RL approaches.

Findings

Achieves over 19% improvement during training.

Attains more than 7% better inference accuracy.

Handles non-Markovian traffic effectively.

Abstract

Reinforcement Learning (RL) has been widely used for packet routing in communication networks, but traditional RL methods rely on the Markov assumption that the current state contains all necessary information for decision-making. In reality, internet traffic is non-Markovian, and past states do influence routing performance. Moreover, common deep RL approaches use function approximators, such as neural networks, that do not model the spatial structure in network topologies. To address these shortcomings, we design a network environment with non-Markovian traffic and introduce a spatial-temporal RL (STRL) framework for packet routing. Our approach outperforms traditional baselines by more than 19% during training and 7% for inference despite a change in network topology.