Imitating Opponent to Win: Adversarial Policy Imitation Learning in Two-player Competitive Games

TL;DR

This paper introduces a novel adversarial policy learning method that uses imitation of the victim's policy to improve effectiveness in two-player competitive games, outperforming existing algorithms.

Contribution

The work proposes a new adversarial policy learning algorithm that incorporates victim policy imitation and feedback, enhancing generalization and effectiveness in multi-agent environments.

Findings

Outperforms state-of-the-art algorithms in MuJoCo games

Leverages imitation learning to better capture victim policy characteristics

Provides theoretical guarantees for imitation accuracy

Abstract

Recent research on vulnerabilities of deep reinforcement learning (RL) has shown that adversarial policies adopted by an adversary agent can influence a target RL agent (victim agent) to perform poorly in a multi-agent environment. In existing studies, adversarial policies are directly trained based on experiences of interacting with the victim agent. There is a key shortcoming of this approach; knowledge derived from historical interactions may not be properly generalized to unexplored policy regions of the victim agent, making the trained adversarial policy significantly less effective. In this work, we design a new effective adversarial policy learning algorithm that overcomes this shortcoming. The core idea of our new algorithm is to create a new imitator to imitate the victim agent's policy while the adversarial policy will be trained not only based on interactions with the victim agent but also based on feedback from the imitator to forecast victim's intention. By doing so, we can leverage the capability of imitation learning in well capturing underlying characteristics of the victim policy only based on sample trajectories of the victim. Our victim imitation learning model differs from prior models as the environment's dynamics are driven by adversary's policy and will keep changing during the adversarial policy training. We provide a provable bound to guarantee a desired imitating policy when the adversary's policy becomes stable. We further strengthen our adversarial policy learning by making our imitator a stronger version of the victim. Finally, our extensive experiments using four competitive MuJoCo game environments show that our proposed adversarial policy learning algorithm outperforms state-of-the-art algorithms.