Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning

TL;DR

This paper introduces a method using successor features to improve coordination among decentralized agents in multi-agent reinforcement learning, especially in unstructured tasks with sparse rewards.

Contribution

It proposes a novel approach to disentangle individual agent impacts on the global value function, enabling stable decentralized training.

Findings

Improved performance over existing methods.

Reduced training time.

Effective in unstructured, sparse reward environments.

Abstract

Multi-agent reinforcement learning (MARL) is a promising framework for solving complex tasks with many agents. However, a key challenge in MARL is defining private utility functions that ensure coordination when training decentralized agents. This challenge is especially prevalent in unstructured tasks with sparse rewards and many agents. We show that successor features can help address this challenge by disentangling an individual agent's impact on the global value function from that of all other agents. We use this disentanglement to compactly represent private utilities that support stable training of decentralized agents in unstructured tasks. We implement our approach using a centralized training, decentralized execution architecture and test it in a variety of multi-agent environments. Our results show improved performance and training time relative to existing methods and suggest that disentanglement of successor features offers a promising approach to coordination in MARL.