Deep Reinforcement Learning for Uplink Scheduling in NOMA-URLLC Networks

TL;DR

This paper introduces NOMA-PPO, a deep reinforcement learning algorithm designed to optimize uplink scheduling for URLLC in NOMA wireless networks, effectively handling strict latency, combinatorial actions, and partial observability.

Contribution

The paper presents a novel DRL-based scheduling algorithm that formulates the problem as a POMDP, adapts PPO for combinatorial actions, and incorporates prior knowledge through a Bayesian policy.

Findings

Outperforms traditional protocols and benchmarks in 3GPP scenarios.

Demonstrates robustness across various channel and traffic conditions.

Efficiently exploits time correlations for improved scheduling.

Abstract

This article addresses the problem of Ultra Reliable Low Latency Communications (URLLC) in wireless networks, a framework with particularly stringent constraints imposed by many Internet of Things (IoT) applications from diverse sectors. We propose a novel Deep Reinforcement Learning (DRL) scheduling algorithm, named NOMA-PPO, to solve the Non-Orthogonal Multiple Access (NOMA) uplink URLLC scheduling problem involving strict deadlines. The challenge of addressing uplink URLLC requirements in NOMA systems is related to the combinatorial complexity of the action space due to the possibility to schedule multiple devices, and to the partial observability constraint that we impose to our algorithm in order to meet the IoT communication constraints and be scalable. Our approach involves 1) formulating the NOMA-URLLC problem as a Partially Observable Markov Decision Process (POMDP) and the introduction of an agent state, serving as a sufficient statistic of past observations and actions, enabling a transformation of the POMDP into a Markov Decision Process (MDP); 2) adapting the Proximal Policy Optimization (PPO) algorithm to handle the combinatorial action space; 3) incorporating prior knowledge into the learning agent with the introduction of a Bayesian policy. Numerical results reveal that not only does our approach outperform traditional multiple access protocols and DRL benchmarks on 3GPP scenarios, but also proves to be robust under various channel and traffic configurations, efficiently exploiting inherent time correlations.