Online RIS Configuration Learning for Arbitrary Large Numbers of $1$-Bit Phase Resolution Elements

TL;DR

This paper introduces novel reinforcement learning methods for efficiently configuring large-scale 1-bit RISs, significantly improving rate maximization by effectively exploring binary action spaces.

Contribution

It develops two variations of DQN and DDPG agents tailored for binary RIS elements, enabling scalable online optimization for large RIS configurations.

Findings

Proposed methods outperform baseline in rate maximization for large RISs.

DQN with Q-function approximation and DDPG with discretization enhance exploration.

Performance comparable to traditional DQN for moderate-sized RISs.

Abstract

Reinforcement Learning (RL) approaches are lately deployed for orchestrating wireless communications empowered by Reconfigurable Intelligent Surfaces (RISs), leveraging their online optimization capabilities. Most commonly, in RL-based formulations for realistic RISs with low resolution phase-tunable elements, each configuration is modeled as a distinct reflection action, resulting to inefficient exploration due to the exponential nature of the search space. In this paper, we consider RISs with 1-bit phase resolution elements, and model the action of each of them as a binary vector including the feasible reflection coefficients. We then introduce two variations of the well-established Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG) agents, aiming for effective exploration of the binary action spaces. For the case of DQN, we make use of an efficient approximation of the Q-function, whereas a discretization post-processing step is applied to the output of DDPG. Our simulation results showcase that the proposed techniques greatly outperform the baseline in terms of the rate maximization objective, when large-scale RISs are considered. In addition, when dealing with moderate scale RIS sizes, where the conventional DQN based on configuration-based action spaces is feasible, the performance of the latter technique is similar to the proposed learning approach.