Variational Offline Multi-agent Skill Discovery

TL;DR

This paper introduces two auto-encoder schemes for automatic multi-agent skill discovery, capturing subgroup and temporal abstractions, enabling transferability across tasks, and improving hierarchical multi-agent reinforcement learning performance.

Contribution

The paper proposes novel auto-encoder schemes that automatically detect subgroups and learn multi-agent skills, addressing a key challenge in multi-agent hierarchical learning.

Findings

Outperforms existing MARL methods on StarCraft tasks.

Skills reduce learning difficulty with delayed and sparse rewards.

Automatically detects latent subgroups based on agent interactions.

Abstract

Skills are effective temporal abstractions established for sequential decision making, which enable efficient hierarchical learning for long-horizon tasks and facilitate multi-task learning through their transferability. Despite extensive research, research gaps remain in multi-agent scenarios, particularly for automatically extracting subgroup coordination patterns in a multi-agent task. In this case, we propose two novel auto-encoder schemes: VO-MASD-3D and VO-MASD-Hier, to simultaneously capture subgroup- and temporal-level abstractions and form multi-agent skills, which firstly solves the aforementioned challenge. An essential algorithm component of these schemes is a dynamic grouping function that can automatically detect latent subgroups based on agent interactions in a task. Further, our method can be applied to offline multi-task data, and the discovered subgroup skills can be transferred across relevant tasks without retraining. Empirical evaluations on StarCraft tasks indicate that our approach significantly outperforms existing hierarchical multi-agent reinforcement learning (MARL) methods. Moreover, skills discovered using our method can effectively reduce the learning difficulty in MARL scenarios with delayed and sparse reward signals. The codebase is available at https://github.com/LucasCJYSDL/VOMASD.