Learning to Play against Any Mixture of Opponents

TL;DR

This paper introduces Q-Mixing, a transfer learning approach that learns Q-values against pure strategies and combines them to perform well against any mixture of opponents, validated in diverse environments.

Contribution

The paper presents Q-Mixing, a novel transfer learning method that constructs policies for mixed opponents from pure-strategy Q-values without additional training.

Findings

Q-Mixing effectively transfers knowledge across opponent mixtures.

Augmenting Q-Mixing with an opponent classifier improves performance and reduces variance.

Q-Mixing performs well in both simple and complex environments.

Abstract

Intuitively, experience playing against one mixture of opponents in a given domain should be relevant for a different mixture in the same domain. We propose a transfer learning method, Q-Mixing, that starts by learning Q-values against each pure-strategy opponent. Then a Q-value for any distribution of opponent strategies is approximated by appropriately averaging the separately learned Q-values. From these components, we construct policies against all opponent mixtures without any further training. We empirically validate Q-Mixing in two environments: a simple grid-world soccer environment, and a complicated cyber-security game. We find that Q-Mixing is able to successfully transfer knowledge across any mixture of opponents. We next consider the use of observations during play to update the believed distribution of opponents. We introduce an opponent classifier -- trained in parallel to Q-learning, using the same data -- and use the classifier results to refine the mixing of Q-values. We find that Q-Mixing augmented with the opponent classifier function performs comparably, and with lower variance, than training directly against a mixed-strategy opponent.