Optimization for Master-UAV-powered Auxiliary-Aerial-IRS-assisted IoT Networks: An Option-based Multi-agent Hierarchical Deep Reinforcement Learning Approach

TL;DR

This paper proposes a hierarchical deep reinforcement learning framework for optimizing collaboration between a master UAV and an auxiliary UAV with IRS in IoT networks, enhancing throughput and energy efficiency.

Contribution

It introduces a novel multi-agent hierarchical DRL approach with option-based learning for UAV cooperation, addressing continuous actions and low hardware requirements.

Findings

CT-MADDPG reduces UAV hardware computational needs.

MADDPOC supports low-level multi-agent cooperative learning.

Proposed methods outperform existing hierarchical DRL approaches.

Abstract

This paper investigates a master unmanned aerial vehicle (MUAV)-powered Internet of Things (IoT) network, in which we propose using a rechargeable auxiliary UAV (AUAV) equipped with an intelligent reflecting surface (IRS) to enhance the communication signals from the MUAV and also leverage the MUAV as a recharging power source. Under the proposed model, we investigate the optimal collaboration strategy of these energy-limited UAVs to maximize the accumulated throughput of the IoT network. Depending on whether there is charging between the two UAVs, two optimization problems are formulated. To solve them, two multi-agent deep reinforcement learning (DRL) approaches are proposed, which are centralized training multi-agent deep deterministic policy gradient (CT-MADDPG) and multi-agent deep deterministic policy option critic (MADDPOC). It is shown that the CT-MADDPG can greatly reduce the requirement on the computing capability of the UAV hardware, and the proposed MADDPOC is able to support low-level multi-agent cooperative learning in the continuous action domains, which has great advantages over the existing option-based hierarchical DRL that only support single-agent learning and discrete actions.