Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning

TL;DR

This paper introduces a model-based reinforcement learning algorithm for multi-agent mean-field control, providing the first regret bounds and demonstrating its effectiveness in large-scale systems with unknown dynamics.

Contribution

It proposes $M^3-UCRL$, a novel algorithm with theoretical guarantees for learning in mean-field control problems with unknown dynamics.

Findings

First regret bounds for model-based RL in MFC

Algorithm effectively balances exploration and exploitation

Flexible with various statistical models like neural networks

Abstract

Learning in multi-agent systems is highly challenging due to several factors including the non-stationarity introduced by agents' interactions and the combinatorial nature of their state and action spaces. In particular, we consider the Mean-Field Control (MFC) problem which assumes an asymptotically infinite population of identical agents that aim to collaboratively maximize the collective reward. In many cases, solutions of an MFC problem are good approximations for large systems, hence, efficient learning for MFC is valuable for the analogous discrete agent setting with many agents. Specifically, we focus on the case of unknown system dynamics where the goal is to simultaneously optimize for the rewards and learn from experience. We propose an efficient model-based reinforcement learning algorithm, $M^3-UCRL$, that runs in episodes, balances between exploration and exploitation during policy learning, and provably solves this problem. Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC, obtained via a novel mean-field type analysis. To learn the system's dynamics, $M^3-UCRL$ can be instantiated with various statistical models, e.g., neural networks or Gaussian Processes. Moreover, we provide a practical parametrization of the core optimization problem that facilitates gradient-based optimization techniques when combined with differentiable dynamics approximation methods such as neural networks.