Distributed Two-tier DRL Framework for Cell-Free Network: Association, Beamforming and Power Allocation

TL;DR

This paper proposes a distributed hierarchical deep reinforcement learning framework for cell-free wireless networks, optimizing spectrum efficiency through two-tier control of association, beamforming, and power allocation, with neural networks operating on different timescales.

Contribution

It introduces a novel two-tier DRL framework with neural networks for hierarchical control in cell-free networks, addressing long-term spectrum efficiency optimization.

Findings

Effective in high-dimensional problems

Improves spectrum efficiency and reduces signaling overhead

Generalizes well to diverse multi-object problems

Abstract

Intelligent wireless networks have long been expected to have self-configuration and self-optimization capabilities to adapt to various environments and demands. In this paper, we develop a novel distributed hierarchical deep reinforcement learning (DHDRL) framework with two-tier control networks in different timescales to optimize the long-term spectrum efficiency (SE) of the downlink cell-free multiple-input single-output (MISO) network, consisting of multiple distributed access points (AP) and user terminals (UT). To realize the proposed two-tier control strategy, we decompose the optimization problem into two sub-problems, AP-UT association (AUA) as well as beamforming and power allocation (BPA), resulting in a Markov decision process (MDP) and Partially Observable MDP (POMDP). The proposed method consists of two neural networks. At the system level, a distributed high-level neural network is introduced to optimize wireless network structure on a large timescale. While at the link level, a distributed low-level neural network is proposed to mitigate inter-AP interference and improve the transmission performance on a small timescale. Numerical results show that our method is effective for high-dimensional problems, in terms of spectrum efficiency, signaling overhead as well as satisfaction probability, and generalize well to diverse multi-object problems.