Multicast Scheduling over Multiple Channels: A Distribution-Embedding Deep Reinforcement Learning Method

TL;DR

This paper introduces a novel deep reinforcement learning method, DE-MAPPO, for multicast scheduling over multiple channels, effectively minimizing energy and latency in complex, constrained MDP environments.

Contribution

It develops a distribution-embedding multi-agent PPO algorithm that handles large action spaces and constraints, and derives an upper performance bound for the problem.

Findings

DE-MAPPO outperforms existing algorithms in numerical tests.

The method achieves performance close to the theoretical upper bound.

It effectively manages large discrete actions and multiple constraints.

Abstract

Multicasting is an efficient technique for simultaneously transmitting common messages from the base station (BS) to multiple mobile users (MUs). Multicast scheduling over multiple channels, which aims to jointly minimize the energy consumption of the BS and the latency of serving asynchronized requests from the MUs, is formulated as an infinite-horizon Markov decision process (MDP) problem with a large discrete action space, multiple time-varying constraints, and multiple time-invariant constraints. To address these challenges, this paper proposes a novel distribution-embedding multi-agent proximal policy optimization (DE-MAPPO) algorithm, which consists of one modified MAPPO and one distribution-embedding module: The former one handles the large discrete action space and time-varying constraints by modifying the structure of the actor networks and the training kernel of the conventional MAPPO; and the latter one iteratively adjusts the action distribution to satisfy the time-invariant constraints. Moreover, a performance upper bound of the considered MDP is derived by solving a two-step optimization problem. Finally, numerical results demonstrate that our proposed algorithm outperforms the existing ones and achieves comparable performance to the derived benchmark.