Reinforcement Learning for Dynamic Resource Allocation in Optical Networks: Hype or Hope?

TL;DR

This paper reviews reinforcement learning for optical network resource management, revealing that simple heuristics often outperform RL solutions and emphasizing the need for better benchmarking and reproducibility.

Contribution

It systematically evaluates benchmark algorithms, recreates landmark problems, and introduces a novel defragmentation-based method to assess network performance limits.

Findings

Simple heuristics outperform RL solutions in tested scenarios.

Benchmark performance is highly sensitive to path selection criteria.

Potential improvements over heuristics are limited to 19-36% increased traffic load.

Abstract

The application of reinforcement learning (RL) to dynamic resource allocation in optical networks has been the focus of intense research activity in recent years, with almost 100 peer-reviewed papers. We present a review of progress in the field, and identify significant gaps in benchmarking practices and reproducibility. To determine the strongest benchmark algorithms, we systematically evaluate several heuristics across diverse network topologies. We find that path count and sort criteria for path selection significantly affect the benchmark performance. We meticulously recreate the problems from five landmark papers and apply the improved benchmarks. Our comparisons demonstrate that simple heuristics consistently match or outperform the published RL solutions, often with an order of magnitude lower blocking probability. Furthermore, we present empirical lower bounds on network blocking using a novel defragmentation-based method, revealing that potential improvements over the benchmark heuristics are limited to 19-36% increased traffic load for the same blocking performance in our examples. We make our simulation framework and results publicly available to promote reproducible research and standardized evaluation https://doi.org/10.5281/zenodo.12594495.