Deep Reinforcement Learning for Energy-Efficient Beamforming Design in Cell-Free Networks

TL;DR

This paper proposes a deep reinforcement learning approach using DDPG to optimize energy-efficient uplink beamforming in cell-free networks, demonstrating convergence to optimal performance through simulations.

Contribution

It introduces a novel DRL-based method for dynamic beamforming design in cell-free networks, leveraging DDPG to maximize long-term energy efficiency.

Findings

DDPG-based beamforming converges to optimal energy efficiency

Hyper-parameter tuning improves EE performance

The method effectively handles continuous state and action spaces

Abstract

Cell-free network is considered as a promising architecture for satisfying more demands of future wireless networks, where distributed access points coordinate with an edge cloud processor to jointly provide service to a smaller number of user equipments in a compact area. In this paper, the problem of uplink beamforming design is investigated for maximizing the long-term energy efficiency (EE) with the aid of deep reinforcement learning (DRL) in the cell-free network. Firstly, based on the minimum mean square error channel estimation and exploiting successive interference cancellation for signal detection, the expression of signal to interference plus noise ratio (SINR) is derived. Secondly, according to the formulation of SINR, we define the long-term EE, which is a function of beamforming matrix. Thirdly, to address the dynamic beamforming design with continuous state and action space, a DRL-enabled beamforming design is proposed based on deep deterministic policy gradient (DDPG) algorithm by taking the advantage of its double-network architecture. Finally, the results of simulation indicate that the DDPG-based beamforming design is capable of converging to the optimal EE performance. Furthermore, the influence of hyper-parameters on the EE performance of the DDPG-based beamforming design is investigated, and it is demonstrated that an appropriate discount factor and hidden layers size can facilitate the EE performance.