Recursive Markov Process for Iterated Games with Markov Strategies

TL;DR

This paper introduces a recursive technique to analyze the stationary distribution of infinite-order Markov processes in iterated games, enabling understanding of complex adaptive strategies with infinite memory.

Contribution

It develops a recursive method to compute the stationary distribution of infinite-order Markov processes in multi-player games, addressing computational challenges.

Findings

The technique successfully analyzes an iterated prisoner's dilemma with infinite memory.

It provides insights into the long-term behavior of adaptive strategies.

The method overcomes limitations of numerical analysis for large k.

Abstract

The dynamics in games involving multiple players, who adaptively learn from their past experience, is not yet well understood. We analyzed a class of stochastic games with Markov strategies in which players choose their actions probabilistically. This class is formulated as a $k^{\text{th}}$ order Markov process, in which the probability of choice is a function of $k$ past states. With a reasonably large $k$ or with the limit $k \to \infty$, numerical analysis of this random process is unfeasible. This study developed a technique which gives the marginal probability of the stationary distribution of the infinite-order Markov process, which can be constructed recursively. We applied this technique to analyze an iterated prisoner's dilemma game with two players who learn using infinite memory.