Reducing AoI and Improving Throughput for NOMA-assisted SGF Systems: A Hierarchical Learning Approach

TL;DR

This paper introduces a hierarchical deep reinforcement learning approach to optimize channel access, beamforming, and scheduling in NOMA-assisted semi-grant-free systems, significantly reducing AoI and enhancing throughput.

Contribution

It proposes a novel hierarchical learning framework combining DRL for transmission scheduling and beamforming, improving system performance over existing methods.

Findings

DRL-based scheduling reduces AoI by up to 5 time slots.

Hierarchical learning achieves approximately 31.82% gain in system efficiency.

Method effective across various GFU and GBU configurations.

Abstract

A non-orthogonal multiple access (NOMA) assisted semi-grant-free (SGF) framework is proposed to enable channel access for grant-free users (GFUs) by using residual resources from grant-based users. Under this framework, the problem of joint beamforming design and transmission scheduling is formulated to improve the system throughput and reduce the age-of-information of GFUs. The aforementioned problem is transferred into a Markov Decision Process to model the changing environment with the transmission/ waiting/ retransmission of GFUs. In an effort to solve the pertinent problem, firstly, a deep reinforcement learning (DRL) based transmission scheduling approach is proposed for determining the optimal transmission probability based on the available transmission slots and transmission status of GFUs. Secondly, a hierarchical learning algorithm is proposed to analyze the channel state information of GBUs and the transmission status of GFUs, and to train an upper-level policy based on this analysis for beamforming to achieve efficient grant-based transmission, while a lower-level policy adapts to maximize the utilization of transmission slots allocated by the upper-level agent. The two policies interact to improve channel access and avoid collisions. Numerical results reveal that 1) The DRL based transmission scheduling outperforms existing adaptive and state-dependent baselines in AoI reduction, where an average three-time-slots-earlier-transmission can be obtained compared to the state-dependent choice, and five time slots earlier can be achieved when comparing to the adaptive choice; 2) The hierarchical learning algorithm is able to achieve approximately a 31.82% gain while maintaining the average AoI of GFUs within 1.5 time slots. 3) The effectiveness of the hierarchical learning scheme in NOMA-assisted SGF system is validated across scenarios with GFUs counts from 1-5 times of GBUs.