Multi-Agent Reinforcement Learning Counteracts Delayed CSI in Multi-Satellite Systems

TL;DR

This paper introduces a multi-agent reinforcement learning algorithm called DS-PPO to improve downlink transmission in multi-satellite systems, effectively handling outdated channel state information and enhancing sum-rate performance.

Contribution

The paper proposes a novel bi-level optimisation algorithm, DS-PPO, for multi-satellite systems to cope with delayed CSI and optimize sum-rate in a distributed multi-antenna setup.

Findings

DS-PPO demonstrates robustness to CSI imperfections.

Significant sum-rate improvements with DS-PPO.

Convergence and computational complexity are analyzed.

Abstract

The integration of satellite communication networks with next-generation (NG) technologies is a promising approach towards global connectivity. However, the quality of services is highly dependant on the availability of accurate channel state information (CSI). Channel estimation in satellite communications is challenging due to the high propagation delay between terrestrial users and satellites, which results in outdated CSI observations on the satellite side. In this paper, we study the downlink transmission of multiple satellites acting as distributed base stations (BS) to mobile terrestrial users. We propose a multi-agent reinforcement learning (MARL) algorithm which aims for maximising the sum-rate of the users, while coping with the outdated CSI. We design a novel bi-level optimisation, procedure themes as dual stage proximal policy optimisation (DS-PPO), for tackling the problem of large continuous action spaces as well as of independent and non-identically distributed (non-IID) environments in MARL. Specifically, the first stage of DS-PPO maximises the sum-rate for an individual satellite and the second stage maximises the sum-rate when all the satellites cooperate to form a distributed multi-antenna BS. Our numerical results demonstrate the robustness of DS-PPO to CSI imperfections as well as the sum-rate improvement attached by the use of DS-PPO. In addition, we provide the convergence analysis for the DS-PPO along with the computational complexity.