Energy-aware optimization of UAV base stations placement via decentralized multi-agent Q-learning

TL;DR

This paper introduces a decentralized multi-agent Q-learning method for UAV base stations that enhances energy efficiency and connectivity in dynamic environments without relying on centralized control.

Contribution

It presents a novel decentralized reinforcement learning approach for UAV placement that outperforms centralized methods in energy and connectivity metrics.

Findings

Decentralized Q-learning improves energy utilization of UAV-BSs.

The method increases the number of connected ground devices.

It outperforms centralized approaches in dynamic scenarios.

Abstract

Unmanned aerial vehicles serving as aerial base stations (UAV-BSs) can be deployed to provide wireless connectivity to ground devices in events of increased network demand, points-of-failure in existing infrastructure, or disasters. However, it is challenging to conserve the energy of UAVs during prolonged coverage tasks, considering their limited on-board battery capacity. Reinforcement learning-based (RL) approaches have been previously used to improve energy utilization of multiple UAVs, however, a central cloud controller is assumed to have complete knowledge of the end-devices' locations, i.e., the controller periodically scans and sends updates for UAV decision-making. This assumption is impractical in dynamic network environments with UAVs serving mobile ground devices. To address this problem, we propose a decentralized Q-learning approach, where each UAV-BS is equipped with an autonomous agent that maximizes the connectivity of mobile ground devices while improving its energy utilization. Experimental results show that the proposed design significantly outperforms the centralized approaches in jointly maximizing the number of connected ground devices and the energy utilization of the UAV-BSs.