Learning Sub-Second Routing Optimization in Computer Networks requires Packet-Level Dynamics

TL;DR

This paper introduces PacketRL, a packet-level reinforcement learning environment for routing, demonstrating that realistic, millisecond-scale network dynamics require packet-level models and novel algorithms for effective sub-second routing optimization.

Contribution

It presents PacketRL, the first packet-level RL environment for routing, and introduces two new algorithms, M-Slim and FieldLines, that outperform existing methods in high-traffic scenarios.

Findings

Packet-level models are necessary for millisecond-scale routing.

Learning strategies trained on fluid models do not generalize well to realistic packet-level environments.

FieldLines achieves millisecond re-optimization without retraining.

Abstract

Finding efficient routes for data packets is an essential task in computer networking. The optimal routes depend greatly on the current network topology, state and traffic demand, and they can change within milliseconds. Reinforcement Learning can help to learn network representations that provide routing decisions for possibly novel situations. So far, this has commonly been done using fluid network models. We investigate their suitability for millisecond-scale adaptations with a range of traffic mixes and find that packet-level network models are necessary to capture true dynamics, in particular in the presence of TCP traffic. To this end, we present $\textit{PackeRL}$, the first packet-level Reinforcement Learning environment for routing in generic network topologies. Our experiments confirm that learning-based strategies that have been trained in fluid environments do not generalize well to this more realistic, but more challenging setup. Hence, we also introduce two new algorithms for learning sub-second Routing Optimization. We present $\textit{M-Slim}$, a dynamic shortest-path algorithm that excels at high traffic volumes but is computationally hard to scale to large network topologies, and $\textit{FieldLines}$, a novel next-hop policy design that re-optimizes routing for any network topology within milliseconds without requiring any re-training. Both algorithms outperform current learning-based approaches as well as commonly used static baseline protocols in scenarios with high-traffic volumes. All findings are backed by extensive experiments in realistic network conditions in our fast and versatile training and evaluation framework.