Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Competitive Games

TL;DR

This paper introduces a novel imitation learning approach integrated with reinforcement learning to improve agent performance in complex multi-agent competitive games, demonstrating superior results over existing methods.

Contribution

It presents a new multi-agent imitation learning model for predicting opponents' moves and a combined RL algorithm, addressing challenges like slow convergence and instability.

Findings

Outperforms state-of-the-art multi-agent RL algorithms in complex environments

Effective in predicting hidden opponent actions and local observations

Achieves superior performance in StarCraft multi-agent challenge environments

Abstract

Training agents in multi-agent competitive games presents significant challenges due to their intricate nature. These challenges are exacerbated by dynamics influenced not only by the environment but also by opponents' strategies. Existing methods often struggle with slow convergence and instability. To address this, we harness the potential of imitation learning to comprehend and anticipate opponents' behavior, aiming to mitigate uncertainties with respect to the game dynamics. Our key contributions include: (i) a new multi-agent imitation learning model for predicting next moves of the opponents -- our model works with hidden opponents' actions and local observations; (ii) a new multi-agent reinforcement learning algorithm that combines our imitation learning model and policy training into one single training process; and (iii) extensive experiments in three challenging game environments, including an advanced version of the Star-Craft multi-agent challenge (i.e., SMACv2). Experimental results show that our approach achieves superior performance compared to existing state-of-the-art multi-agent RL algorithms.