Metric Policy Representations for Opponent Modeling

TL;DR

This paper introduces a novel approach to opponent modeling in multi-agent reinforcement learning by learning policy representations that encode similarities and differences, enabling better generalization to unseen agents.

Contribution

The paper proposes a general method for learning policy representations that reflect policy similarities and differences, improving generalization in multi-agent environments.

Findings

Representations encode policy similarities effectively.

Agents conditioned on learned representations generalize to unseen agents.

Method outperforms existing approaches in multi-agent tasks.

Abstract

In multi-agent reinforcement learning, the inherent non-stationarity of the environment caused by other agents' actions posed significant difficulties for an agent to learn a good policy independently. One way to deal with non-stationarity is opponent modeling, by which the agent takes into consideration the influence of other agents' policies. Most existing work relies on predicting other agents' actions or goals, or discriminating between different policies. However, such modeling fails to capture the similarities and differences between policies simultaneously and thus cannot provide enough useful information when generalizing to unseen agents. To address this, we propose a general method to learn representations of other agents' policies, such that the distance between policies is deliberately reflected by the distance between representations, while the policy distance is inferred from the sampled joint action distributions during training. We empirically show that the agent conditioned on the learned policy representation can well generalize to unseen agents in three multi-agent tasks.