MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrations

TL;DR

MisoDICE introduces a two-stage approach for offline multi-agent imitation learning from unlabeled mixed-quality demonstrations, combining trajectory labeling with a novel multi-agent IL algorithm to improve policy robustness.

Contribution

The paper proposes MisoDICE, a new multi-agent imitation learning algorithm that extends DICE with value decomposition, enabling effective learning from unlabeled mixed-quality data.

Findings

MisoDICE outperforms existing methods on standard benchmarks.

Effective trajectory labeling improves imitation quality.

Robust policies are learned even with scarce expert data.

Abstract

We study offline imitation learning (IL) in cooperative multi-agent settings, where demonstrations have unlabeled mixed quality - containing both expert and suboptimal trajectories. Our proposed solution is structured in two stages: trajectory labeling and multi-agent imitation learning, designed jointly to enable effective learning from heterogeneous, unlabeled data. In the first stage, we combine advances in large language models and preference-based reinforcement learning to construct a progressive labeling pipeline that distinguishes expert-quality trajectories. In the second stage, we introduce MisoDICE, a novel multi-agent IL algorithm that leverages these labels to learn robust policies while addressing the computational complexity of large joint state-action spaces. By extending the popular single-agent DICE framework to multi-agent settings with a new value decomposition and mixing architecture, our method yields a convex policy optimization objective and ensures consistency between global and local policies. We evaluate MisoDICE on multiple standard multi-agent RL benchmarks and demonstrate superior performance, especially when expert data is scarce.