Communication-Efficient Orchestrations for URLLC Service via Hierarchical Reinforcement Learning

TL;DR

This paper introduces a hierarchical reinforcement learning framework for URLLC in 5G, enabling multi-level decision-making with reduced delay and signaling overhead, improving resource allocation efficiency.

Contribution

The paper proposes a novel multi-agent Hierarchical RL framework that operates at different control loop timescales for URLLC, enhancing performance and reducing overhead.

Findings

Outperforms single-agent RL in factory automation scenarios

Reduces signaling overhead and delay significantly

Optimizes retransmissions and transmission power effectively

Abstract

Ultra-reliable low latency communications (URLLC) service is envisioned to enable use cases with strict reliability and latency requirements in 5G. One approach for enabling URLLC services is to leverage Reinforcement Learning (RL) to efficiently allocate wireless resources. However, with conventional RL methods, the decision variables (though being deployed at various network layers) are typically optimized in the same control loop, leading to significant practical limitations on the control loop's delay as well as excessive signaling and energy consumption. In this paper, we propose a multi-agent Hierarchical RL (HRL) framework that enables the implementation of multi-level policies with different control loop timescales. Agents with faster control loops are deployed closer to the base station, while the ones with slower control loops are at the edge or closer to the core network providing high-level guidelines for low-level actions. On a use case from the prior art, with our HRL framework, we optimized the maximum number of retransmissions and transmission power of industrial devices. Our extensive simulation results on the factory automation scenario show that the HRL framework achieves better performance as the baseline single-agent RL method, with significantly less overhead of signal transmissions and delay compared to the one-agent RL methods.