Hierarchical Deep Double Q-Routing

TL;DR

This paper presents a hierarchical deep reinforcement learning method using DDQN for scalable, adaptive, and efficient packet routing in large, dynamic communication networks, balancing end-to-end performance with local resource management.

Contribution

It introduces a hierarchical cluster-based DDQN routing algorithm that effectively manages end-to-end performance and local network resources in large, dynamic networks.

Findings

Scales well in large networks

Adapts to dynamic network demands

Improves resource utilization

Abstract

This paper explores a deep reinforcement learning approach applied to the packet routing problem with high-dimensional constraints instigated by dynamic and autonomous communication networks. Our approach is motivated by the fact that centralized path calculation approaches are often not scalable, whereas the distributed approaches with locally acting nodes are not fully aware of the end-to-end performance. We instead hierarchically distribute the path calculation over designated nodes in the network while taking into account the end-to-end performance. Specifically, we develop a hierarchical cluster-oriented adaptive per-flow path calculation mechanism by leveraging the Deep Double Q-network (DDQN) algorithm, where the end-to-end paths are calculated by the source nodes with the assistance of cluster (group) leaders at different hierarchical levels. In our approach, a deferred composite reward is designed to capture the end-to-end performance through a feedback signal from the source nodes to the group leaders and captures the local network performance through the local resource assessments by the group leaders. This approach scales in large networks, adapts to the dynamic demand, utilizes the network resources efficiently and can be applied to segment routing.