Hierarchical Deep Reinforcement Learning for Age-of-Information Minimization in IRS-aided and Wireless-powered Wireless Networks

TL;DR

This paper introduces a hierarchical deep reinforcement learning framework to optimize transmission scheduling and control strategies in IRS-assisted wireless-powered sensor networks, significantly reducing age-of-information and improving fairness.

Contribution

It proposes a novel hierarchical DRL approach for joint scheduling and control in IRS-aided wireless networks with dynamic data arrivals, enhancing AoI minimization.

Findings

Hierarchical DRL outperforms baseline methods in AoI reduction.

The framework improves proportional fairness among nodes.

An efficient approximation reduces inner-loop computation time.

Abstract

In this paper, we focus on a wireless-powered sensor network coordinated by a multi-antenna access point (AP). Each node can generate sensing information and report the latest information to the AP using the energy harvested from the AP's signal beamforming. We aim to minimize the average age-of-information (AoI) by adapting the nodes' transmission scheduling and the transmission control strategies jointly. To reduce the transmission delay, an intelligent reflecting surface (IRS) is used to enhance the channel conditions by controlling the AP's beamforming vector and the IRS's phase shifting matrix. Considering dynamic data arrivals at different sensing nodes, we propose a hierarchical deep reinforcement learning (DRL) framework to for AoI minimization in two steps. The users' transmission scheduling is firstly determined by the outer-loop DRL approach, e.g. the DQN or PPO algorithm, and then the inner-loop optimization is used to adapt either the uplink information transmission or downlink energy transfer to all nodes. A simple and efficient approximation is also proposed to reduce the inner-loop rum time overhead. Numerical results verify that the hierarchical learning framework outperforms typical baselines in terms of the average AoI and proportional fairness among different nodes.