Global Behavior of Learning Dynamics in Zero-Sum Games with Memory Asymmetry

TL;DR

This paper analyzes the global behavior of learning dynamics in zero-sum games with asymmetric memory, revealing conditions for convergence to Nash equilibrium using novel conserved and Lyapunov quantities.

Contribution

It introduces new analytical tools, including an extended Kullback-Leibler divergence and Lyapunov functions, to characterize complex learning dynamics with asymmetric memory.

Findings

Strategies converge to Nash when X exploits Y

Y's deviation from equilibrium increases X's exploitability

Global convergence to Nash is supported by theoretical and experimental evidence

Abstract

This study examines the global behavior of dynamics in learning in games between two players, X and Y. We consider the simplest situation for memory asymmetry between two players: X memorizes the other Y's previous action and uses reactive strategies, while Y has no memory. Although this memory complicates their learning dynamics, we characterize the global behavior of such complex dynamics by discovering and analyzing two novel quantities. One is an extended Kullback-Leibler divergence from the Nash equilibrium, a well-known conserved quantity from previous studies. The other is a family of Lyapunov functions of X's reactive strategy. One of the global behaviors we capture is that if X exploits Y, then their strategies converge to the Nash equilibrium. Another is that if Y's strategy is out of equilibrium, then X becomes more exploitative with time. Consequently, we suggest global convergence to the Nash equilibrium from both aspects of theory and experiment. This study provides a novel characterization of the global behavior in learning in games through a couple of indicators.