Structured Imitation Learning of Interactive Policies through Inverse Games

TL;DR

This paper introduces a structured imitation learning framework that combines generative policy learning with game-theoretic modeling to improve interactive multi-agent policies from demonstrations.

Contribution

It presents a novel two-step approach that separates behavioral pattern learning from inter-agent dependency modeling via inverse game solving.

Findings

Significantly improves non-interactive policies in multi-agent tasks.

Performs comparably to ground truth interactive policies with only 50 demonstrations.

Demonstrates potential of structured imitation learning in interactive multi-agent settings.

Abstract

Generative model-based imitation learning methods have recently achieved strong results in learning high-complexity motor skills from human demonstrations. However, imitation learning of interactive policies that coordinate with humans in shared spaces without explicit communication remains challenging, due to the significantly higher behavioral complexity in multi-agent interactions compared to non-interactive tasks. In this work, we introduce a structured imitation learning framework for interactive policies by combining generative single-agent policy learning with a flexible yet expressive game-theoretic structure. Our method explicitly separates learning into two steps: first, we learn individual behavioral patterns from multi-agent demonstrations using standard imitation learning; then, we structurally learn inter-agent dependencies by solving an inverse game problem. Preliminary results in a synthetic 5-agent social navigation task show that our method significantly improves non-interactive policies and performs comparably to the ground truth interactive policy using only 50 demonstrations. These results highlight the potential of structured imitation learning in interactive settings.