Simulation-Driven Reinforcement Learning in Queuing Network Routing Optimization

TL;DR

This paper introduces a simulation-driven reinforcement learning framework using DDPG and Dyna planning for optimizing routing in complex queueing networks, demonstrating improved stability, efficiency, and scalability in dynamic environments.

Contribution

It presents a novel Dyna-DDPG approach with predictive models for queueing network routing, enhancing robustness and sample efficiency over traditional methods.

Findings

Effective routing policies learned rapidly under disruptions

Framework scales well to larger networks

Demonstrates robustness in uncertain environments

Abstract

This study focuses on the development of a simulation-driven reinforcement learning (RL) framework for optimizing routing decisions in complex queueing network systems, with a particular emphasis on manufacturing and communication applications. Recognizing the limitations of traditional queueing methods, which often struggle with dynamic, uncertain environments, we propose a robust RL approach leveraging Deep Deterministic Policy Gradient (DDPG) combined with Dyna-style planning (Dyna-DDPG). The framework includes a flexible and configurable simulation environment capable of modeling diverse queueing scenarios, disruptions, and unpredictable conditions. Our enhanced Dyna-DDPG implementation incorporates separate predictive models for next-state transitions and rewards, significantly improving stability and sample efficiency. Comprehensive experiments and rigorous evaluations demonstrate the framework's capability to rapidly learn effective routing policies that maintain robust performance under disruptions and scale effectively to larger network sizes. Additionally, we highlight strong software engineering practices employed to ensure reproducibility and maintainability of the framework, enabling practical deployment in real-world scenarios.