Inverse Factorized Q-Learning for Cooperative Multi-agent Imitation Learning

TL;DR

This paper introduces a novel inverse factorized Q-learning algorithm for cooperative multi-agent imitation learning, effectively handling high-dimensional spaces and agent dependencies, and demonstrating superior performance in complex multi-agent environments.

Contribution

The paper proposes a new multi-agent imitation learning method using mixing networks for centralized training, with theoretical convexity conditions and extensive experimental validation.

Findings

Outperforms existing multi-agent IL algorithms in challenging environments

Successfully learns local and joint value functions in multi-agent settings

Demonstrates effectiveness on complex environments like SMACv2

Abstract

This paper concerns imitation learning (IL) (i.e, the problem of learning to mimic expert behaviors from demonstrations) in cooperative multi-agent systems. The learning problem under consideration poses several challenges, characterized by high-dimensional state and action spaces and intricate inter-agent dependencies. In a single-agent setting, IL has proven to be done efficiently through an inverse soft-Q learning process given expert demonstrations. However, extending this framework to a multi-agent context introduces the need to simultaneously learn both local value functions to capture local observations and individual actions, and a joint value function for exploiting centralized learning. In this work, we introduce a novel multi-agent IL algorithm designed to address these challenges. Our approach enables the centralized learning by leveraging mixing networks to aggregate decentralized Q functions. A main advantage of this approach is that the weights of the mixing networks can be trained using information derived from global states. We further establish conditions for the mixing networks under which the multi-agent objective function exhibits convexity within the Q function space. We present extensive experiments conducted on some challenging competitive and cooperative multi-agent game environments, including an advanced version of the Star-Craft multi-agent challenge (i.e., SMACv2), which demonstrates the effectiveness of our proposed algorithm compared to existing state-of-the-art multi-agent IL algorithms.