MARL-GPT: Foundation Model for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces MARL-GPT, a transformer-based model trained via offline reinforcement learning to perform well across diverse multi-agent environments without task-specific tuning.

Contribution

It presents a unified, multi-task MARL model using a single GPT architecture trained on large expert datasets, enabling broad applicability.

Findings

MARL-GPT achieves competitive performance across multiple environments.

The model requires no task-specific tuning.

Training on large datasets enables generalization across tasks.

Abstract

Recent advances in multi-agent reinforcement learning (MARL) have demonstrated success in numerous challenging domains and environments, but typically require specialized models for each task. In this work, we propose a coherent methodology that makes it possible for a single GPT-based model to learn and perform well across diverse MARL environments and tasks, including StarCraft Multi-Agent Challenge, Google Research Football and POGEMA. Our method, MARL-GPT, applies offline reinforcement learning to train at scale on the expert trajectories (400M for SMACv2, 100M for GRF, and 1B for POGEMA) combined with a single transformer-based observation encoder that requires no task-specific tuning. Experiments show that MARL-GPT achieves competitive performance compared to specialized baselines in all tested environments. Thus, our findings suggest that it is, indeed, possible to build a multi-task transformer-based model for a wide variety of (significantly different) multi-agent problems paving the way to the fundamental MARL model (akin to ChatGPT, Llama, Mistral etc. in natural language modeling).