A Deep Policy Inference Q-Network for Multi-Agent Systems

TL;DR

This paper introduces DPIQN and DRPIQN, deep reinforcement learning models that infer other agents' policies to improve decision-making in multi-agent systems, demonstrating superior performance in competitive and cooperative scenarios.

Contribution

The paper proposes novel deep policy inference Q-networks that incorporate inferred policy features, enhancing multi-agent learning under varying strategies and partial observability.

Findings

DPIQN and DRPIQN outperform baseline DQN and DRQN in soccer simulations.

Models adapt well to dynamic policy changes of collaborators and opponents.

Enhanced stability and higher mean scores achieved in multi-agent tasks.

Abstract

We present DPIQN, a deep policy inference Q-network that targets multi-agent systems composed of controllable agents, collaborators, and opponents that interact with each other. We focus on one challenging issue in such systems---modeling agents with varying strategies---and propose to employ "policy features" learned from raw observations (e.g., raw images) of collaborators and opponents by inferring their policies. DPIQN incorporates the learned policy features as a hidden vector into its own deep Q-network (DQN), such that it is able to predict better Q values for the controllable agents than the state-of-the-art deep reinforcement learning models. We further propose an enhanced version of DPIQN, called deep recurrent policy inference Q-network (DRPIQN), for handling partial observability. Both DPIQN and DRPIQN are trained by an adaptive training procedure, which adjusts the network's attention to learn the policy features and its own Q-values at different phases of the training process. We present a comprehensive analysis of DPIQN and DRPIQN, and highlight their effectiveness and generalizability in various multi-agent settings. Our models are evaluated in a classic soccer game involving both competitive and collaborative scenarios. Experimental results performed on 1 vs. 1 and 2 vs. 2 games show that DPIQN and DRPIQN demonstrate superior performance to the baseline DQN and deep recurrent Q-network (DRQN) models. We also explore scenarios in which collaborators or opponents dynamically change their policies, and show that DPIQN and DRPIQN do lead to better overall performance in terms of stability and mean scores.