To be a fast adaptive learner: using game history to defeat opponents

TL;DR

This paper introduces F3, a novel framework that leverages past game history to enable AI agents to quickly adapt and outperform opponents with unknown strategies in repeated game scenarios.

Contribution

The paper proposes a new framework (F3) with an Opponent Action Estimator that effectively fuses past and current game history for rapid opponent modeling.

Findings

F3-trained agents outperform deep reinforcement learning agents in fixed turns.

F3 enables quick adaptation to unknown opponent strategies.

OAE module contains transferable meta-knowledge across different games.

Abstract

In many real-world games, such as traders repeatedly bargaining with customers, it is very hard for a single AI trader to make good deals with various customers in a few turns, since customers may adopt different strategies even the strategies they choose are quite simple. In this paper, we model this problem as fast adaptive learning in the finitely repeated games. We believe that past game history plays a vital role in such a learning procedure, and therefore we propose a novel framework (named, F3) to fuse the past and current game history with an Opponent Action Estimator (OAE) module that uses past game history to estimate the opponent's future behaviors. The experiments show that the agent trained by F3 can quickly defeat opponents who adopt unknown new strategies. The F3 trained agent obtains more rewards in a fixed number of turns than the agents that are trained by deep reinforcement learning. Further studies show that the OAE module in F3 contains meta-knowledge that can even be transferred across different games.