Pure Strategy Best Responses to Mixed Strategies in Repeated Games

TL;DR

This paper demonstrates that in repeated games, particularly the iterated prisoner's dilemma, pure strategy best responses to mixed strategies can be computed efficiently, but such responses may not exist in multi-agent tournaments.

Contribution

It introduces a method to compute pure strategy best responses in repeated games using SMT or MDP solvers and generalizes the results to k-memory strategies.

Findings

Pure strategy best responses exist for one-memory strategies.

Such responses can be computed efficiently with SMT or MDP solvers.

Pure strategy best responses may not exist in multi-agent tournaments.

Abstract

Repeated games are difficult to analyze, especially when agents play mixed strategies. We study one-memory strategies in iterated prisoner's dilemma, then generalize the result to k-memory strategies in repeated games. Our result shows that there always exists a pure strategy best response, which can be computed with SMT or MDP solvers. However, there may not exist such pure strategy best response in multi-agent tournaments. All source code is released for verification.

Figures

Tables

Table 1. Table 1: Prisoner’s Dilemma

	c	d
c	(R,R)	(S,T)
d	(T,S)	(P,P)

Table 2. Table 2: Outcome and Strategy

Outcome∗	$cc$	$cd$	$dc$	$dd$
Strategy of $𝐩$	$p_1$	$p_2$	$p_3$	$p_4$
Strategy of $𝐪$	$q_1$	$q_3$	$q_2$	$q_4$

Equations

$$s_i=\underset{k\to \infty }{\lim }\frac{{\sum }_{j=1}^k{s}_i^{(j)}}{k}$$

$$\mathbf{M}=\left(\begin{array}{cccc}{p}_1{q}_1& p_1(1-q_1)& (1-p_1)q_1& (1-p_1)(1-q_1)\\ p_2{q}_3& p_2(1-q_3)& (1-p_2)q_3& (1-p_2)(1-q_3)\\ p_3{q}_2& p_3(1-q_2)& (1-p_3)q_2& (1-p_3)(1-q_2)\\ p_4{q}_4& p_4(1-q_4)& (1-p_4)q_4& (1-p_4)(1-q_4)\end{array}\right)$$

$$\mathbf{M}{\mathbf{v}}^r={\mathbf{v}}^{r+1}$$

$$\mathbf{Mv}=\mathbf{v}$$

$$D(\mathbf{p},\mathbf{q},\mathbf{f})=\left|\begin{array}{cccc}{p}_1{q}_1-1& p_1-1& q_1-1& f_1\\ p_2{q}_3& p_2-1& q_3& f_2\\ p_3{q}_2& p_3& q_2-1& f_3\\ p_4{q}_4& p_4& q_4& f_4\end{array}\right|$$

$$s_X=\frac{D(\mathbf{p},\mathbf{q},{\mathbf{S}}_{\mathbf{X}})}{D(\mathbf{p},\mathbf{q},1)},s_Y=\frac{D(\mathbf{p},\mathbf{q},{\mathbf{S}}_{\mathbf{Y}})}{D(\mathbf{p},\mathbf{q},1)}$$

$$\displaystyle\big{(}\bigwedge_{i=1,2,3,4}0\leq p_{i}\leq 1\big{)}\quad\land\quad\big{(}\bigwedge_{j=1,2,3,4}0<q_{j}<1\big{)}$$

$$\displaystyle\land\quad\big{(}\mathbf{p}\neq(1,1,0,0)\big{)}\quad\land\quad\big{(}D(\mathbf{p},\mathbf{q},\mathbf{1})\geq 0\big{)}$$

$$s_X=\frac{D(\mathbf{p},\mathbf{q},{\mathbf{S}}_{\mathbf{X}})}{D(\mathbf{p},\mathbf{q},1)}$$

$$M=D(\mathbf{p},\mathbf{q},{\mathbf{S}}_{\mathbf{X}}),N=D(\mathbf{p},\mathbf{q},1)$$

$$s_{X}={M}\big{/}{N}$$

$$\frac{\partial s_{X}}{\partial p_{1}}=\frac{\partial M\big{/}\partial p_{1}\cdot N-\partial N\big{/}\partial p_{1}\cdot M}{N^{2}}$$

$$U=\partial M\big{/}\partial p_{1}\cdot N-\partial N\big{/}\partial p_{1}\cdot M$$

$${\partial U}\big{/}{\partial p_{1}}=0$$

$$F_{15}=\frac{-R\ast q_3+S\ast q_1-S}{q_1-q_3-1}$$

$$F_0=F_4=F_8,F_{13}=F_{14}=F_{15}$$

$$(0,0,0,0)\equiv (0,1,0,0)\equiv (1,0,0,0)$$

$$(1,1,0,1)\equiv (1,1,1,0)\equiv (1,1,1,1)$$

$$\mathcal{F}=\{F_0,F_1,F_2,F_3,F_5,F_6,F_7,F_9,F_{10},F_{11},F_{15}\}$$

$$\mathcal{K}\mathcal{B}\equiv$$

$$\wedge (T>R>P>S)\wedge (2R>T+S)$$

$$\mathcal{K}\mathcal{B}\wedge (q_3\le q_1\le q_4\le q_2)\supset \underset{F^{\prime }\in \mathcal{F}\backslash \{F_0\}}{\bigwedge }{F}_0>F^{\prime }$$

$$\lnot\big{(}\mathcal{KB}\land(q_{3}\leq q_{1}\leq q_{4}\leq q_{2})\big{)}\lor\bigwedge_{F^{\prime}\in\mathcal{F}\backslash\{F_{0}\}}F_{0}>F^{\prime}$$

$$\big{(}\mathcal{KB}\land(q_{3}\leq q_{1}\leq q_{4}\leq q_{2})\big{)}\land\bigvee_{F^{\prime}\in\mathcal{F}\backslash\{F_{0}\}}F_{0}\leq F^{\prime}$$

$$\bigvee_{F^{\prime}\in\mathcal{F}\backslash\{F_{0}\}}\big{(}\mathcal{KB}\land(q_{3}\leq q_{1}\leq q_{4}\leq q_{2})\land F_{0}\leq F^{\prime}\big{)}$$

$$\mathcal{MC}\equiv$$

$$\overline{q_3}=\frac{(1-q_1)(P-S)+q_4(R-S)}{R-P}$$

$$\mathcal{K}\mathcal{B}\wedge \mathcal{MC}\wedge (q_2=\overline{q_2})\wedge (q_3=\overline{q_3})\supset \underset{F_i,F_j\in \mathcal{F}}{\bigwedge }{F}_i=F_j$$

$$\mathcal{K}\mathcal{B}\wedge \mathcal{MC}\wedge (q_2<\overline{q_2})\wedge (q_3=\overline{q_3})\supset \underset{F^{\prime }\in \mathcal{F}\backslash \{F_{15}\}}{\bigwedge }{F}_{15}>F^{\prime }$$

$$\mathcal{K}\mathcal{B}\wedge \mathcal{MC}\wedge (q_2=\overline{q_2})\wedge (q_3>\overline{q_3})\supset \underset{F^{\prime }\in \mathcal{F}\backslash \{F_0\}}{\bigwedge }{F}_0>F^{\prime }$$

$$\mathcal{E}\mathcal{X}\equiv$$

$$\wedge$$

Full text

1 Introduction

Repeated games are hard to analyze. It’s well known that Nash equilibrium is the solution concept in one-shot games, whereas there exist infinite Nash equilibria in repeated games according to the folk theorem. In most repeated games, a player responses to the previous actions of the other player, thus the player’s strategy need to be analysed with dynamic game theory Han (2018). When the game is infinitely repeated, one cannot simply add up the payoff of each round, which in general will be infinite. Average rewards or discounted rewards are usually used to evaluate a strategy, and we focus on the former in this paper.

In repeated games, a player’s strategy is a function from histories of interactions to actions. Sometimes one restricts strategies to some specific forms, such as Turing machines Chen and Tang (2015); Knoblauch (1994); Megiddo and Wigderson (1986), finite automata Rubinstein (1986); Ben-Porath (1990); Gilboa (1988); Zuo and Tang (2015), ones with limited memories  Hauert and Schuster (1997); Lindgren (1992); Chen et al. (2017), and other forms of bounded rationality (e.g. Osborne and Rubinstein (1994); Shoham and Leyton-Brown (2008)).

A mixed strategy maps histories to a probability distribution of actions. Press and Dyson (2012) concluded that shortest memory sets the rule of the game, then Chen et al. (2017) pointed out the best response to a k-memory strategy in infinitely repeated games should also be k-memory. We further explore the best response to mixed strategies. Our result shows that there always exists a pure strategy best response in two-agent infinitely repeated games, but it’s not the case in multi-agent tournaments.

In the first place we study a concrete repeated game, namely the iterated prisoner’s dilemma (IPD). It involves two agents playing repeatedly the Prisoner’s Dilemma (PD) in table 1. In the PD, each player can choose between Cooperate (c) and Defect (d). If both choose c, they receive a payoff of $R$ (rewards); If both choose d, they receive a payoff of $P$ (penalty); If one chooses c and the other d, the defector receives a payoff of $T$ (temptation to defect) and the cooperator receives a payoff of $S$ (sucker’s payoff). The assumption is that $T>R>P>S$ and that $2*R>T+S$, which makes $(d,d)$ the only equilibrium in one-shot game, but cooperation provides more utility in the long run.

When both agents take one-memory mixed strategies, the IPD can be modeled as a Markov chain, whose stationary distribution can be computed according to Press and Dyson (2012) and the best response can be solved using an SMT solver like Z3 De Moura and Bjørner (2008). We give a method to compute the best response and then summarize the best response to some popular strategies. In the meanwhile, our analysis explains the behavior of evolutionary agents. The solves a problem in Press and Dyson (2012) as we can now formally prove the behaviour of these agents.

Some of our results on one-memory strategies can be generalized to k-memory mixed strategies. In order to compute the best response to a completely mixed strategy, we build a Markov decision process (MDP) and compute its optimal policy. Later we prove that this is a communicating MDP, which has a pure optimal policy independent of the initial state. This shows that there always exists a pure strategy best response to any completely mixed strategy in a repeated game.

However, this result does not hold for tournaments with more than two agents. We use a similar Markov chain model to compute the best strategy in a tournament of several one-memory agents, and show that the best response can not be a pure strategy.

In this paper, when a strategy $\mathbf{q}$ is given, we calculate its best response $\mathbf{p}$. Most calculations or proofs are too laboursome to be done manually, so we show how state-of-art computer solvers assist us in solving this classic economic problem and bringing us new insights. In most cases, our program solved the problems in seconds. We release all our source code for verification.

The rest of this paper is organized as follows. First we study one-memory strategies of the iterated prisoner’s dilemma. Then we model k-memory problems into MDP and solve it with existing algorithms. Later we analyze a multi-agent tournament which serves as a counter example. Finally we give some discussion and concluding remarks.

2 One-Memory Strategies in IPD

2.1 One-Memory Strategies

In the section we consider a concrete example where both players play one-memory strategies in the iterated prisoner’s dilemma (IPD).

The PD game in Table 1 is played by two players X and Y for infinite rounds. The score (or payoff) of either player is calculated by average score in the limit (cf. Shoham and Leyton-Brown (2008)). Given an infinite sequence of scores $s_{i}^{(1)},s_{i}^{(2)},...$ for player $i\in\{X,Y\}$, the average score of $i$ is

[TABLE]

Although a player can have unlimited memory and decide what to do based on the entire history of the interactions so far, one-memory strategies base their response only on the outcome of the previous round. Press and Dyson Press and Dyson (2012) proved that shortest-memory player sets the rules of the game, in the sense that, for any strategy of the longer-memory player Y, X’s score is exactly the same as if Y had played a certain shorter memory strategy, disregarding any history in excess of that shared with X. This conclusion enables us to focus on one-memory strategies.

A one-memory strategy consists of an initial state $p_{0}$ (the probability to cooperate in the first round) and a vector $\textbf{p}=(p_{1},p_{2},p_{3},p_{4})=(p_{cc},p_{cd},p_{dc},p_{dd})$ where $p_{z}$ is the probability of playing Cooperate when the outcome $z$ occurred in the previous round.

If X uses the initial probability $p_{0}$ and strategy $\textbf{p}=(p_{1},p_{2},p_{3},p_{4})$, Y uses the initial probability $q_{0}$ and strategy $\textbf{q}=(q_{1},q_{2},q_{3},q_{4})$, then the probability distribution of the first iteration is $\textbf{v}^{1}=(p_{0}q_{0},p_{0}(1-q_{0}),(1-p_{0})q_{0},(1-p_{0})(1-q_{0}))$ and the successive outcomes follow a Markov chain with transition matrix given by:

[TABLE]

The probability distribution in r-th iteration $\mathbf{v}^{r}$ over the set of outcomes is a non-negative vector with unit sum, indexed by four states, $\mathbf{v}^{r}=(v_{cc}^{r},v_{cd}^{r},v_{dc}^{r},v_{dd}^{r})=(v_{1}^{r},v_{2}^{r},v_{3}^{r},v_{4}^{r})$.

Notice that we define outcome from each player’s perspective, for example, if the outcome is $cd$ from X’s perspective, it is $dc$ from Y’s. $\mathbf{v}$ is defined from player X’s perspective, meaning that $v_{cd}$ refers to X plays C and Y plays D. If the previous outcome is $cd$, X’s probability of cooperation is $p_{2}$ while Y’s is $p_{3}$. See Table 2 for this correspondence.

Thus each entry of $\mathbf{M}$ represents the probability of transition between different states, which satisfies

[TABLE]

In accordance with Akin (2016), we will call $\mathbf{M}$ convergent when $\mathbf{M}$ has a unique stationary distribution of $\mathbf{v}$, which satisfies,

[TABLE]

Press and Dyson (2012) gave a determinant representation of player’s payoff when it is calculated by the limit of average. For an arbitrary four-vector $\mathbf{f}=(f_{1},f_{2},f_{3},f_{4})$ , let

[TABLE]

Then the average payoff of $s_{X}$ and $s_{Y}$ can be calculated with payoff vector $\mathbf{S_{X}}=(R,S,T,P)$ and $\mathbf{S_{Y}}=(R,T,S,P)$.

[TABLE]

2.2 Ergodic Markov Chain

According to Akin (2016), the following statements on stationary distribution are equivalent.

• •

There is a unique stationary distribution $\mathbf{v}$ in accordance with $\mathbf{M}$.

• •

The stationary distribution $\mathbf{v}$ is independent of the initial distribution $\mathbf{v}^{1}$, and thus $p_{0}$ and $q_{0}$.

• •

There is no absorbing states (trapped states) in the Markov chain.

• •

$D(\mathbf{p},\mathbf{q},\mathbf{1})\neq 0$

To avoid $D(\mathbf{p},\mathbf{q},\mathbf{1})=0$, we assume $\mathbf{q}$ plays a completely mixed strategy, that is, $q_{i}\in(0,1)$. For the convenience of defining best response, we assume $p_{i}\in[0,1]$ and $\mathbf{p}\neq(1,1,0,0)$ (namely strategy Repeat). The last assumption makes sense because if $\mathbf{p}$ plays $Cooperate$ in the first round, strategy Repeat is equivalent to Always Cooperate, and otherwise to Always Defect. Under these assumptions, we can prove $D(\mathbf{p},\mathbf{q},\mathbf{1})\neq 0$ with the SMT solver Z3 De Moura and Bjørner (2008).

Theorem 1**.**

Assume $p_{i}\in[0,1]$ and $q_{i}\in(0,1)$, and that $\mathbf{p}\neq(1,1,0,0)$, then $D(\mathbf{p},\mathbf{q},\mathbf{1})<0$.

Proof.

The negation of Theorem 1 is, $\exists\mathbf{p},\mathbf{q}$

[TABLE]

Z3 returns “unsatisfiable” to (2.2), meaning that its value is always False. Thus Theorem 1 is always True. ∎

In practice we accelerate this proof with domain knowledge. We first prove that $D(\mathbf{p},\mathbf{q},\mathbf{1})$ is monotonic to $p_{i}$ by calculating derivatives(see also proof of Thm 2), then calculate extrema by letting $p_{i}\in\{0,1\}$. Since all extrema are less than zero, the determinant is less than zero.

2.3 Symbolic Calculation

Theorem 2** (Monotonicity).**

*Suppose $p_{i}\in[0,1]$ and $q_{i}\in(0,1)$, $\mathbf{p}\neq(1,1,0,0)$, $s_{X},s_{Y}$ are defined in equation (4), and $z\in\{p_{1},...,p_{4},q_{1},...,q_{4}\}$.

When all variables in $\{p_{1},...,p_{4},q_{1},...,q_{4}\}\backslash\{z\}$ are fixed, $s_{X}$ is monotonic to $z$.*

Proof.

According to Theorem 1, $D(\mathbf{p},\mathbf{q},\mathbf{1})\neq 0$.

[TABLE]

For simplicity, let

[TABLE]

Then

[TABLE]

Without loss of generality, get partial derivative to $p_{1}$,

[TABLE]

Let

[TABLE]

Again, get partial derivative of $U$ to $p_{1}$,

[TABLE]

In Eq. (6), since $N^{2}>0$, and the numerator is not a function of $p_{1}$, we can draw the conclusion that, when $\{p_{2},p_{3},p_{4},q_{1},q_{2},q_{3},q_{4}\}$ are fixed, $s_{X}$ is monotonic in terms of $p_{1}$.

Similarly, $s_{X}$ is also monotonic to any variable in $p_{i},q_{i}$ when other variables are fixed. ∎

As symbolic calculations of determinant are laboursome, we use a Python library SymPy Meurer et al. (2017) and double checked with MATLAB.

Given a fixed completely mixed strategy $\mathbf{q}$, we want to compute its best response $\mathbf{p}$. Theorem 2 implies that there exists a pure one-memory strategy best response. We only need to enumerate all pure strategy $\mathbf{p}$, and one of them must be a best response to $\mathbf{q}$.

Theorem 3**.**

There always exists an one-memory pure strategy best response to completely mixed one-memory strategies.

Press and Dyson (2012) did some experiments to show that evolutionary players who evolve to get better payoff will finally reach the optimal reward but they didn’t prove it analytically. Actually this property follows from Theorem 2. No matter where a strategy begins, it will finally evolve to the best response. In other words, there is no local maxima.

We are now able to compute all extreme values of $s_{X}$ by letting $p_{i}=0$ or $p_{i}=1$. The result is denoted as $F_{k}$, where $k=\sum_{i=1}^{4}p_{i}*2^{4-i}$. For instance, when $\mathbf{p}=(1,1,1,1)$,

[TABLE]

Some other $F_{k}$ has a much more complex expression. We’ll not show them due to limited space but readers can refer to our source code for details. Notice that $F_{12}$ is not considered as it corresponds to Repeat strategy, $\mathbf{p}=(1,1,0,0)$. Some formulas turn out to have the same value, that is,

[TABLE]

The corresponding strategies always receive the same payoff whatever the co-player’s strategy is, we call such strategies equivalent.

Theorem 4** (Equivalent).**

When playing agaisnt completely mixed strategies, the following strategies are equivalent.

[TABLE]

Duplicates will not be considered in the rest of this paper, and the set of distinct expressions is defined as,

[TABLE]

2.4 Theorem Discovery

With concrete values of (R,S,T,P), when strategy $\mathbf{q}$ is given, we can simply compute all values in $\mathcal{F}$ so that the strategy corresponding to the largest value is a best response. Our experiments show that concrete values of (R,S,T,P) really matter. When the concrete setting of the IPD varies, the best responses to some strategies also change. But some general theorems are discovered with SMT solver Z3.

The knowledge base is defined according to the constraints of the IPD and mixed one-memory strategies. In this section, $\supset$ means “logical imply” , $\equiv$ means “logical equivalent” and $\lnot$ means “logical not”. Theorems are proved by refutation.

Definition 1** (Knowledge Base).**

[TABLE]

If a strategy is less cooperative under a better situation, there is no reason to cooperate with him. As $T>R>P>S$ is the setting of the prisoner’s dilemma, column player’s preference is $CD\succ CC\succ DD\succ DC$. If his strategy $\mathbf{q}$ satisfies $q_{3}\leq q_{1}\leq q_{4}\leq q_{2}$, then there is no chance of reciprocity. See Table 2 for this correspondence.

Theorem 5**.**

If strategy $\mathbf{q}=\{q_{1},q_{2},q_{3},q_{4}\}$ satisfies $q_{3}\leq q_{1}\leq q_{4}\leq q_{2}$, then Always Defect is its best response .

Proof.

First, we give a formal representation of this theorem. $\forall\mathbf{p},\mathbf{q},R,S,T,P$,

[TABLE]

Eq. (9) is equivalent to, $\forall\mathbf{p},\mathbf{q},R,S,T,P$,

[TABLE]

This theorem can be proved by conclusion refutation, and the negation of Eq. (10) is, $\exists\mathbf{p},\mathbf{q},R,S,T,P$,

[TABLE]

which is equivalent to,

[TABLE]

Z3 returns “unsatisfiable” to equation (12), implying that its negation(i.e. Theorem 5) is always true. ∎

A natural corollary of Theorem 5 is, if $q_{1}=q_{2}=q_{3}=q_{4}$, then Always Defect is a best response. One famous example of such strategy is Random, i.e. $\mathbf{q}=(0.5,0.5,0.5,0.5)$. This corollary illustrates that if some player is indifferent to the outcome of previous round, there is no reason to cooperate with him. More discussion on this property and backward inducation can be found in Section 4.3.

Some other interesting theorems are discovered on the neighbor of Mischief strategy defined by Press and Dyson (2012). Such strategy is also called equalizer. We only show the theorems together with their formal representations as the proofs are very similar.

Definition 2** (MisChief).**

A MisChief strategy is an one-memory strategy $\mathbf{q}=(q_{1},\overline{q_{2}},\overline{q_{3}},q_{4})$ defined as,

[TABLE]

Theorem 6** (Mischief).**

While playing with MisChief strategy $\mathbf{q}=(q_{1},q_{2},q_{3},q_{4})$, every strategy of $\mathbf{p}$ receives the same average payoff. Formally, $\forall\mathbf{p},\mathbf{q},R,S,T,P$

[TABLE]

Theorem 7** (MisTort).**

While playing with strategy $\mathbf{q}=(q_{1},q_{2},q_{3},q_{4})$ s.t. $q_{2}<\overline{q_{2}}$, $q_{3}=\overline{q_{3}}$, Always Cooperate is a best response. Formally, $\forall\mathbf{p},\mathbf{q},R,S,T,P$,

[TABLE]

Theorem 8**.**

While playing with strategy $\mathbf{q}=(q_{1},q_{2},q_{3},q_{4})$ s.t. $q_{2}=\overline{q_{2}}$,$q_{3}>\overline{q_{3}}$, Always Defect is a best response. Formally, $\forall\mathbf{p},\mathbf{q},R,S,T,P$,

[TABLE]

We call the strategy in Theorem 7 MisTort because it’s on the neighbor of MisChief strategy and shares the property of extortionate strategies in Press and Dyson (2012), to which the best response is Always Cooperate. Due to similar property, readers may guess that one may be the a subclass of the other. However, there turns out to be no intersection between MisTort and extortionate strategies.

Theorem 9**.**

There is no intersection between extortionate strategy and MisTort strategy.

Proof.

Extortionate strategy($\mathcal{EX}$) is defined in Press and Dyson (2012),

[TABLE]

The negation of Theorem 9 is, $\exists\mathbf{q}$,

[TABLE]

Z3 returns “unsat”, meaning Theorem 9 is always true. ∎

One specific instance of MisTort strategy can be calculated by letting $q_{1}=0.9,q_{4}=0.1$ and $(R,S,T,P)=(3,0,5,1)$. From Eq. (7) we can calculate out $q_{2}<0.7,q_{3}=0.2$. A possible MisTort strategy is, $\mathbf{q}=(0.9,0.5,0.2,0.1)$. We enumerate a number of strategy $\mathbf{p}$ to play against this $\mathbf{q}=(0.9,0.5,0.2,0.1)$, and the average payoff of both players is shown in Figure 1. All pairs of payoffs $(s_{X},s_{Y})$ form a compact convex hull, and the case that $\mathbf{p}=(1,1,1,1)$ is receives highest payoff. The figure implies that Always Cooperate is a best response to $\mathbf{q}$, leading to a payoff where $s_{X}=2.0,s_{Y}=3.67$.

3 Repeated Games and MDP

3.1 Repeated Game and K-memory Strategy

A finite, 2-person normal form game is a tuple $(N,A,u)$, where

• •

$N=\{1,2\}$ is the set of two players.

• •

$A=A_{1}\times A_{2}$, where $A_{i}$ is a finite set of actions available to player $i$. Each vector $a=(a_{1},a_{2})\in A$ is called an action profile (or outcome).

• •

$u=(u_{1},u_{2})$, where $u_{i}:A\mapsto\mathbb{R}$ is a real-valued utility (or payoff) function for player i.

This stage game is played for infinite rounds. A player’s payoff is defined as the average payoff of the stage game in the limit Shoham and Leyton-Brown (2008). Similar to Eq. (1), given an infinite sequence of payoffs $u_{i}^{(1)}$, $u_{i}^{(2)}$,… for player i, the average payoff of $i$ is \lim_{k\to\infty}{\sum_{j=1}^{k}u_{i}^{(j)}}\big{/}{k}.

A k-memory mixed strategy of player $i$ is a function that maps k action profiles to a distribution on actions,

[TABLE]

Given previous k actions profiles $s\in A^{k}$, the probability of taking action $a_{i}\in A_{i}$ is written as $p_{i}(s,a_{i})$. In the start of the game, there aren’t k action profiles, in this case we fill empty action profile with action $c$. Our conclusions don’t rely on the outcomes of the first k rounds, see Thm 11.

3.2 Markov Decision Process

Now we compute the best response to a k-memory strategy. The first question is how much memory is required to be a best response. According to Press and Dyson (2012), shortest memory sets the rule of the game, so that it needs to be at most k-memory. On the other hand, any strategy with less than k-memory can be represented by a k-memory strategy. Then we know k-memory strategy should be exactly enough. This problem is also discussed in Chen et al. (2017).

Both players take k-memory strategies. Assume player 2 plays a completely mixed strategy $p_{2}$, namely $\forall s\in S,a_{2}\in A_{2},p_{2}(s,a_{2})>0$, then from player 1’s point of view, his action and payoff can be modeled as the following MDP.

• •

Actions: same as $A_{1}$ in the repeated game.

• •

States: $S=A^{k}$, the set of vectors of $k$ action profiles, indexed by 1,2,…,k, from farthest to most recent. Any $s_{1:k}\in S$ refers to k continuous action profiles. When there comes a new action profile $(a_{1},a_{2})$, it goes to a new state $s^{\prime}=s_{2:k}+(a_{1},a_{2})$, which drops the farthest action profile $s_{1}$ and appends $(a_{1},a_{2})$.

• •

Transition Function. When player 1 takes action $a_{1}$, the states transits from state $s$ to $s^{\prime}=s_{2:k}+(a_{1},a_{2})$ with probability $T(s,a_{1},s^{\prime})=p_{2}(s,a_{2})$. At every state $s$, for any action $a_{1}$, $\sum_{s^{\prime}}T(s,a_{1},s^{\prime})=1$.

• •

Reward $R(s,a_{1})=u_{1}(s_{k}),\forall a_{1}\in A_{1}$, meaning player 1’s utility of the most recent action profile.

A mixed policy $\pi$ assigns a probability distribution of actions to each state. Formally, $\pi:S\mapsto\Delta(A_{1})$. For each $s\in S$, the probability of taking action $a_{1}\in A_{1}$ is written as $\pi(s,a_{1})$. A completely mixed policy is $\pi(s,a_{1})>0,\forall s\in S,a_{1}\in A_{1}$. $\Pi^{+}$ is the set of all completely mixed policies, which is a subset of all policies $\Pi$. A pure policy is that, for each $s\in S$, there is exactly one $a_{1}\in A_{1}$ satisfying $\pi(s,a_{1})=1$. From the definition of state $S$ and strategy in Eq. (15), we can easily see that every policy $\pi$ corresponds to a strategy $p_{1}$.

Player 1 walks among states for infinite rounds, starting from any state $s\in S$. At each round, it takes an action according to current state and its policy $\pi$, then it goes into a new state. This agent receives a sequence of rewards, namely $R^{1},R^{2},...,R^{j},...$, and the final score $\rho^{\pi}(s)$ is calculated by the expectation of average, formally,

[TABLE]

Our goal is to find an optimal policy $\pi^{*}$ that maximizes the expectation of average payoff, $\forall\pi,\rho^{\pi^{*}}(s)\geq\rho^{\pi}(s)$. We will show that the initial state $s$ doesn’t matter, for our MDP.

3.3 Solving Average-Payoff MDPs

There is a detailed explanation on average-payoff MDPs in textbook Puterman (2014). Our results in this part are mainly based on propositions and theorems in Filar and Schultz (1988), which are introduced as lemmas. To be specific, Lemma 1 is from paragraph 2, section 1; Definition 3 is from Definition 1.1, and Lemma 2 is from Theorem 2.1 in Filar and Schultz (1988).

Lemma 1**.**

While computing optimal policy, it is sufficient to consider pure policies.

A policy $\pi$ induces a Markov chain on states $S$ with transition matrix $\mathbf{M}(\pi)$, whose entries $\mathbf{M}_{st}(\pi)$ denote the probability of transition from state s to state t when policy $\pi$ is followed. For $\tau$ being a nonnegative integer, $\mathbf{M}^{\tau}(\pi)$ denotes the $\tau$-th power of square matrix $\mathbf{M}(\pi)$, and $\mathbf{M}^{\tau}_{st}(\pi)$ denotes an entry in it.

Definition 3**.**

An MDP is communicating if, for every pair of states $s,t\in S$, there exists a pure policy $\pi$ and an integer $\tau\geq 1$ such that $\mathbf{M}^{\tau}_{st}(\pi)$ is strictly positive.

Lemma 2**.**

Let $\Pi^{+}$ be the set of completely mixed policies. The following two conditions are equivalent.

• •

An MDP is communicating.

• •

Every policy $\pi^{+}\in\Pi^{+}$ induces an irreducible $\mathbf{M}(\pi^{+})$.

Since we have assumed that player 2 takes a completely mixed strategy $p_{2}$, for every $\pi^{+}\in\Pi^{+}$, the induced matrix $\mathbf{M}(\pi^{+})$ is irreducible. In other words, when both players take completely mixed strategies, there is no transient state in the corresponding Markov chain, because it’s possible to append any action profile to any state.

Theorem 10**.**

The MDP in section 3.2 is communicating. Its optimal policy can be calculated by linear programming, which is independent of starting state. If $\pi^{*}$ is an optimal policy starting from state $s$, then it is also an optimal policy starting from any other state $s^{\prime}$.

Then there is a theorem in repeated games.

Theorem 11**.**

There always exists a k-memory pure strategy best response to k-memory completely mixed strategy in infinitely repeated games, which is independent of the initial k outcomes.

Such pure strategy best response can be computed with existing MDP solvers, e.g. MDPtoolbox Chadès et al. (2014). As an example, we compute the best response to Stochastic Tif-for-2-Tats(STF2T). The game is IPD in Table 1 with (R,S,T,P)=(3,0,5,1). STF2T cooperates with probability 0.1 when the other player defects for continuous two rounds, and cooperates with probability 0.9 otherwise. MDPtoolbox solves this model, the best response is to play $c$ and $d$ alternatively, and the average payoff is 2.67. See our source code for details.

4 Discussion

4.1 Multi-agent Tournament

In the previous sections we have discussed how to calculate the best response in two-agent repeated games. Our result mainly relies on the fact that there always exists a pure strategy best response. Now we consider the best response in multi-agent tournaments, which may not be a pure strategy.

Due to the complexity of symbolic calculations, we conduct experiments of a multi-agent IPD tournament instead of giving an analytical proof. Consider a tournament of eleven one-memory agents, namely one $\mathbf{p}$, nine $\mathbf{q}$ and one $\mathbf{u}$. Suppose $(R,S,T,P)=(3,0,5,1)$, $\mathbf{q}=(0.9,0.5,0.2,0.1)$ and $\mathbf{u}=(0.4,0.8,0.2,0.6)$, we want to compute the optimal $\mathbf{p}$ that maximizes his payoff. Strategy $\mathbf{p}$ plays with every $\mathbf{q}$ and $\mathbf{u}$ respectively and the final score takes the average of all games, which can be calculated according to Eq. (4),

[TABLE]

In two-agent games, there always exists a pure strategy best response. As is shown in Theorem 7 and Theorem 5, the best response to $\mathbf{q}$ is Always Cooperate, while the best response to $\mathbf{u}$ is Always Defect. Now we consider whether there is a pure strategy one-memory best response $\mathbf{p}^{*}$ in this tournament .

First we let $\mathbf{p}$ be a pure strategy and compute the values of $s_{\mathbf{p}}$. There are $2^{4}-1=15$ pure strategies, where Repeat strategy $\mathbf{p}=(1,1,0,0)$ is excluded. Among them, strategy Tit-for-Tat $\mathbf{p}=(1,0,1,0)$ receives highest payoff of 1.90. However, after we consider mixed strategies, we found that strategy $\mathbf{p}=(1,0.9,0,0.1)$ reaches a higher payoff of 2.02.

This specific example shows that there may not exist a pure strategy best response in general multi-agent tournaments.

4.2 Limitation of Z3

Although Z3 is effective in solving linear constraints and successfully prove our theorems, it sometimes fails to give a solution in several hours when the constraint is a non-linear combination of several variables. We take comprehensive measures to overcome this challenge. (1) Avoid quantifiers. A mixture of $\forall$ and $\exists$ prevents us from efficient proof. We avoid using both quantifiers and manually simplify some formulas. (2) Simplify formulas. We take advantage of domain knowledge to remove fractions, such as in the proof of Theorem 1 we first prove monotonicity and prove all extreme values are less than zero. (3) Break down. We solve one clause of conjunction or disjunction at one time. Usually a theorems requires every clause has the same value of true or false, such as Eq. (12). (4) Remove variables. When SMT cannot solve a theorem, we have to replace some variables with concrete values to get some conjectures or a weaker theorem. In section 4.1 we compute a counter-example instead of prove it analytically.

4.3 Backward Induction

There has been many debates on backward induction Kreps et al. (1982)Binmore (1997). For finite n round IPD, playing defect is the best strategy in the one-shot game at n-th round because there is no further possibility of reciprocity. Since both rational players will play defect in the n-th round, there is no reason to cooperate in the (n-1)-th round. By backward induction, Always Defect is the only equilibrium of finite iterated prisoner’s dilemma.

To explain the emergence of cooperation, it is usually assumed that the game is played for infinite rounds, or an unknown number of roundsAxelrod and Hamilton (1981). Although this assumption invalidates backward induction, there still lacks an explanation of cooperation. Actually, when people conduct backward induction, they implicitly assume that the outcome of previous round has no effect on the decision of current round. According to the corollary of Theorem 5, when someone is indifferent to the outcome of previous round, there is no reason to cooperate with him. Therefore, we hold the view that the emergence of cooperation cannot be explained without taking previous history into consideration.

5 Conclusion

In this paper we compute best responses to mixed strategies in repeated games. Our main result shows that there always exists a pure strategy best response in two-agent repeated games. Based on this result, we analyze one-memory strategies, give the method to compute best response, and discover new theorems in the iterated prisoner’s dilemma. The work enhances our comprehension of the IPD and explains the evolutionary behavior left over in Press and Dyson (2012).

We generalize this result to the best response to k-memory strategies. Such problem is modeled as MDP and solved with existing solvers. In a multi-agent tournament, however, there may not exist a pure strategy best response. As a result, computing the best strategy in a tournament where an agent should take the same strategy to all other agents is still an open question.

As most calculations and proofs are conducted by computer programs, we release all our source code for verification. Source code is uploaded as additional files.