Towards Practical Operation of Deep Reinforcement Learning Agents in Real-World Network Management at Open RAN Edges

TL;DR

This paper presents a practical framework for deploying deep reinforcement learning in real-world network management, addressing key challenges like traffic variability, heterogeneity, and convergence in Open RAN environments.

Contribution

It introduces an MEC-O-RAN orchestration framework and a comprehensive solution strategy for deploying DRL in live networks, bridging the gap between theory and practice.

Findings

Validated on urban testing infrastructure

Demonstrated handling of asynchronous traffic

Reduced convergence time with transfer learning

Abstract

Deep Reinforcement Learning (DRL) has emerged as a powerful solution for meeting the growing demands for connectivity, reliability, low latency and operational efficiency in advanced networks. However, most research has focused on theoretical analysis and simulations, with limited investigation into real-world deployment. To bridge the gap and support practical DRL deployment for network management, we first present an orchestration framework that integrates ETSI Multi-access Edge Computing (MEC) with Open RAN, enabling seamless adoption of DRL-based strategies across different time scales while enhancing agent lifecycle management. We then identify three critical challenges hindering DRL's real-world deployment, including (1) asynchronous requests from unpredictable or bursty traffic, (2) adaptability and generalization across heterogeneous topologies and evolving service demands, and (3) prolonged convergence and service interruptions due to exploration in live operational environments. To address these challenges, we propose a three-fold solution strategy: (a) advanced time-series integration for handling asynchronized traffic, (b) flexible architecture design such as multi-agent DRL and incremental learning to support heterogeneous scenarios, and (c) simulation-driven deployment with transfer learning to reduce convergence time and service disruptions. Lastly, the feasibility of the MEC-O-RAN architecture is validated on an urban-wide testing infrastructure, and two real-world use cases are presented, showcasing the three identified challenges and demonstrating the effectiveness of the proposed solutions.