What game are we playing? End-to-end learning in normal and extensive form games

TL;DR

This paper introduces a differentiable, end-to-end learning framework for inferring unknown game parameters from observations, enabling neural networks to learn and solve complex normal and extensive form games like poker and security scenarios.

Contribution

It develops a primal-dual Newton method and a backpropagation approach to compute gradients through game solutions, facilitating end-to-end learning of game parameters.

Findings

Effective in poker and security game tasks

Enables learning game parameters from observations

Integrates differentiable game solvers into deep networks

Abstract

Although recent work in AI has made great progress in solving large, zero-sum, extensive-form games, the underlying assumption in most past work is that the parameters of the game itself are known to the agents. This paper deals with the relatively under-explored but equally important "inverse" setting, where the parameters of the underlying game are not known to all agents, but must be learned through observations. We propose a differentiable, end-to-end learning framework for addressing this task. In particular, we consider a regularized version of the game, equivalent to a particular form of quantal response equilibrium, and develop 1) a primal-dual Newton method for finding such equilibrium points in both normal and extensive form games; and 2) a backpropagation method that lets us analytically compute gradients of all relevant game parameters through the solution itself. This ultimately lets us learn the game by training in an end-to-end fashion, effectively by integrating a "differentiable game solver" into the loop of larger deep network architectures. We demonstrate the effectiveness of the learning method in several settings including poker and security game tasks.