Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors

TL;DR

This paper introduces Bidirectional Distillation, a novel framework for multi-agent reinforcement learning that enhances generalization to unseen co-players by combining forward and reverse knowledge distillation, without extensive policy storage.

Contribution

It proposes a new mixed-play framework, Bidirectional Distillation, that improves policy generalization in MARL by leveraging alternating distillation directions and avoiding costly policy storage.

Findings

BiDist significantly improves generalization across various tasks.

It diversifies policy distribution space effectively.

Empirical results support theoretical analysis of BiDist's effectiveness.

Abstract

Population-population generalization is a challenging problem in multi-agent reinforcement learning (MARL), particularly when agents encounter unseen co-players. However, existing self-play-based methods are constrained by the limitation of inside-space generalization. In this study, we propose Bidirectional Distillation (BiDist), a novel mixed-play framework, to overcome this limitation in MARL. BiDist leverages knowledge distillation in two alternating directions: forward distillation, which emulates the historical policies' space and creates an implicit self-play, and reverse distillation, which systematically drives agents towards novel distributions outside the known policy space in a non-self-play manner. In addition, BiDist operates as a concise and efficient solution without the need for the complex and costly storage of past policies. We provide both theoretical analysis and empirical evidence to support BiDist's effectiveness. Our results highlight its remarkable generalization ability across a variety of cooperative, competitive, and social dilemma tasks, and reveal that BiDist significantly diversifies the policy distribution space. We also present comprehensive ablation studies to reinforce BiDist's effectiveness and key success factors. Source codes are available in the supplementary material.