Hierarchical Imitation Learning of Team Behavior from Heterogeneous Demonstrations

TL;DR

This paper introduces DTIL, a hierarchical multi-agent imitation learning algorithm that effectively learns complex, multimodal team behaviors from heterogeneous demonstrations, improving over existing methods in collaborative tasks.

Contribution

The paper presents DTIL, a hierarchical MAIL algorithm capable of modeling diverse team behaviors from heterogeneous data, addressing limitations of prior single-policy assumptions.

Findings

DTIL outperforms MAIL baselines in various collaborative scenarios.

DTIL accurately models multimodal team behaviors.

The method scales well to long horizons and continuous states.

Abstract

Successful collaboration requires team members to stay aligned, especially in complex sequential tasks. Team members must dynamically coordinate which subtasks to perform and in what order. However, real-world constraints like partial observability and limited communication bandwidth often lead to suboptimal collaboration. Even among expert teams, the same task can be executed in multiple ways. To develop multi-agent systems and human-AI teams for such tasks, we are interested in data-driven learning of multimodal team behaviors. Multi-Agent Imitation Learning (MAIL) provides a promising framework for data-driven learning of team behavior from demonstrations, but existing methods struggle with heterogeneous demonstrations, as they assume that all demonstrations originate from a single team policy. Hence, in this work, we introduce DTIL: a hierarchical MAIL algorithm designed to learn multimodal team behaviors in complex sequential tasks. DTIL represents each team member with a hierarchical policy and learns these policies from heterogeneous team demonstrations in a factored manner. By employing a distribution-matching approach, DTIL mitigates compounding errors and scales effectively to long horizons and continuous state representations. Experimental results show that DTIL outperforms MAIL baselines and accurately models team behavior across a variety of collaborative scenarios.