Reinforcement Learning of Beam Codebooks in Millimeter Wave and Terahertz MIMO Systems

TL;DR

This paper introduces a deep reinforcement learning framework that autonomously optimizes beamforming codebooks in mmWave and terahertz MIMO systems, reducing training overhead and adapting to environmental and hardware variations without explicit channel knowledge.

Contribution

It proposes a novel RL-based method with a Wolpertinger-variant architecture for efficient beam pattern optimization in large antenna arrays, respecting hardware constraints.

Findings

Learns near-optimal beam patterns in LOS and non-LOS scenarios

Operates effectively without explicit channel knowledge

Adapts to hardware impairments and environment variations

Abstract

Millimeter wave (mmWave) and terahertz MIMO systems rely on pre-defined beamforming codebooks for both initial access and data transmission. Being pre-defined, however, these codebooks are commonly not optimized for specific environments, user distributions, and/or possible hardware impairments. This leads to large codebook sizes with high beam training overhead which increases the initial access/tracking latency and makes it hard for these systems to support highly mobile applications. To overcome these limitations, this paper develops a deep reinforcement learning framework that learns how to iteratively optimize the codebook beam patterns (shapes) relying only on the receive power measurements and without requiring any explicit channel knowledge. The developed model learns how to autonomously adapt the beam patterns to best match the surrounding environment, user distribution, hardware impairments, and array geometry. Further, this approach does not require any knowledge about the channel, array geometry, RF hardware, or user positions. To reduce the learning time, the proposed model designs a novel Wolpertinger-variant architecture that is capable of efficiently searching for an optimal policy in a large discrete action space, which is important for large antenna arrays with quantized phase shifters. This complex-valued neural network architecture design respects the practical RF hardware constraints such as the constant-modulus and quantized phase shifter constraints. Simulation results based on the publicly available DeepMIMO dataset confirm the ability of the developed framework to learn near-optimal beam patterns for both line-of-sight (LOS) and non-LOS scenarios and for arrays with hardware impairments without requiring any channel knowledge.