Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a novel offline multi-agent reinforcement learning framework that leverages shared good trajectories and attention mechanisms to improve policy learning in diverse data quality scenarios.

Contribution

The paper proposes the Shared Individual Trajectories (SIT) framework, utilizing attention-based reward decomposition and graph attention networks to enhance offline MARL performance.

Findings

Significantly improved results in complex offline multi-agent datasets.

Effective handling of data quality disparities among individual trajectories.

Demonstrated success in both discrete and continuous control environments.

Abstract

Offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets, which is an important step toward the deployment of multi-agent systems in real-world applications. However, in practice, each individual behavior policy that generates multi-agent joint trajectories usually has a different level of how well it performs. e.g., an agent is a random policy while other agents are medium policies. In the cooperative game with global reward, one agent learned by existing offline MARL often inherits this random policy, jeopardizing the performance of the entire team. In this paper, we investigate offline MARL with explicit consideration on the diversity of agent-wise trajectories and propose a novel framework called Shared Individual Trajectories (SIT) to address this problem. Specifically, an attention-based reward decomposition network assigns the credit to each agent through a differentiable key-value memory mechanism in an offline manner. These decomposed credits are then used to reconstruct the joint offline datasets into prioritized experience replay with individual trajectories, thereafter agents can share their good trajectories and conservatively train their policies with a graph attention network (GAT) based critic. We evaluate our method in both discrete control (i.e., StarCraft II and multi-agent particle environment) and continuous control (i.e, multi-agent mujoco). The results indicate that our method achieves significantly better results in complex and mixed offline multi-agent datasets, especially when the difference of data quality between individual trajectories is large.