Strategies that enforce linear payoff relationships under observation errors in Repeated Prisoner's Dilemma game

TL;DR

This paper analytically investigates strategies in the repeated prisoner's dilemma with observation errors, revealing that only zero-determinant and unconditional strategies can enforce linear payoff relationships despite errors.

Contribution

It extends the understanding of linear payoff enforcement strategies to scenarios with observation errors, identifying the limited strategy sets capable of such enforcement.

Findings

Only ZD and unconditional strategies enforce linear payoffs with errors.

Numerical confirmation supports the analytical results.

Observation errors do not enable new strategies to enforce linear relationships.

Abstract

The theory of repeated games analyzes the long-term relationship of interacting players and mathematically reveals the condition of how cooperation is achieved, which is not achieved in a one-shot game. In the repeated prisoner's dilemma (RPD) game with no errors, zero-determinant (ZD) strategies allow a player to unilaterally set a linear relationship between the player's own payoff and the opponent's payoff regardless of the strategy that the opponent implements. In contrast, unconditional strategies such as ALLD and ALLC also unilaterally set a linear payoff relationship. Errors often happen between players in the real world. However, little is known about the existence of such strategies in the RPD game with errors. Here, we analytically search for all strategies that enforce a linear payoff relationship under observation errors in the RPD game. As a result, we found that, even in the case with observation errors, the only strategy sets that enforce a linear payoff relationship are either ZD strategies or unconditional strategies and that no other strategies can enforce it, which were numerically confirmed.

Figures

Equations

$$\bordermatrix{&g&b\cr{\rm C}&R&S\cr{\rm D}&T&P\cr}.$$

$$f_i(a)=\sum _{\omega }{u}_i(a_i,{\omega }_i)\sigma (\omega |a).$$

$$T_E>R_E>P_E>S_E,$$

$$2R_E>T_E+S_E,$$

$$\mathbf{p}=(p_1,p_2,p_3,p_4),$$

$$\mathbf{q}=(q_1,q_2,q_3,q_4),$$

$$M=\scalebox{0.85}{$\displaystyle\left(\begin{array}[]{ll}\left(\begin{array}[]{ll}\tau p_{1}q_{1}\\ +\epsilon p_{1}q_{2}\\ +\epsilon p_{2}q_{1}\\ +\xi p_{2}q_{2}\end{array}\right)\left(\begin{array}[]{ll}\tau p_{1}(1-q_{1})\\ +\epsilon p_{1}(1-q_{2})\\ +\epsilon p_{2}(1-q_{1})\\ +\xi p_{2}(1-q_{2})\end{array}\right)\left(\begin{array}[]{ll}\tau(1-p_{1})q_{1}\\ +\epsilon(1-p_{1})q_{2}\\ +\epsilon(1-p_{2})q_{1}\\ +\xi(1-p_{2})q_{2}\end{array}\right)\left(\begin{array}[]{ll}\tau(1-p_{1})(1-q_{1})\\ +\epsilon(1-p_{1})(1-q_{2})\\ +\epsilon(1-p_{2})(1-q_{1})\\ +\xi(1-p_{2})(1-q_{2})\end{array}\right)\\ \left(\begin{array}[]{ll}\epsilon p_{1}q_{3}\\ +\xi p_{1}q_{4}\\ +\tau p_{2}q_{3}\\ +\epsilon p_{2}q_{4}\end{array}\right)\left(\begin{array}[]{ll}\epsilon p_{1}(1-q_{3})\\ +\xi p_{1}(1-q_{4})\\ +\tau p_{2}(1-q_{3})\\ +\epsilon p_{2}(1-q_{4})\end{array}\right)\left(\begin{array}[]{ll}\epsilon(1-p_{1})q_{3}\\ +\xi(1-p_{1})q_{4}\\ +\tau(1-p_{2})q_{3}\\ +\epsilon(1-p_{2})q_{4}\end{array}\right)\left(\begin{array}[]{ll}\epsilon(1-p_{1})(1-q_{3})\\ +\xi(1-p_{1})(1-q_{4})\\ +\tau(1-p_{2})(1-q_{3})\\ +\epsilon(1-p_{2})(1-q_{4})\end{array}\right)\\ \left(\begin{array}[]{ll}\epsilon p_{3}q_{1}\\ +\tau p_{3}q_{2}\\ +\xi p_{4}q_{1}\\ +\epsilon p_{4}q_{2}\end{array}\right)\left(\begin{array}[]{ll}\epsilon p_{3}(1-q_{1})\\ +\tau p_{3}(1-q_{2})\\ +\xi p_{4}(1-q_{1})\\ +\epsilon p_{4}(1-q_{2})\end{array}\right)\left(\begin{array}[]{ll}\epsilon(1-p_{3})q_{1}\\ +\tau(1-p_{3})q_{2}\\ +\xi(1-p_{4})q_{1}\\ +\epsilon(1-p_{4})q_{2}\end{array}\right)\left(\begin{array}[]{ll}\epsilon(1-p_{3})(1-q_{1})\\ +\tau(1-p_{3})(1-q_{2})\\ +\xi(1-p_{4})(1-q_{1})\\ +\epsilon(1-p_{4})(1-q_{2})\end{array}\right)\\ \left(\begin{array}[]{ll}\xi p_{3}q_{3}\\ +\epsilon p_{3}q_{4}\\ +\epsilon p_{4}q_{3}\\ +\tau p_{4}q_{4}\end{array}\right)\left(\begin{array}[]{ll}\xi p_{3}(1-q_{3})\\ +\epsilon p_{3}(1-q_{4})\\ +\epsilon p_{4}(1-q_{3})\\ +\tau p_{4}(1-q_{4})\end{array}\right)\left(\begin{array}[]{ll}\xi(1-p_{3})q_{3}\\ +\epsilon(1-p_{3})q_{4}\\ +\epsilon(1-p_{4})q_{3}\\ +\tau(1-p_{4})q_{4}\end{array}\right)\left(\begin{array}[]{ll}\xi(1-p_{3})(1-q_{3})\\ +\epsilon(1-p_{3})(1-q_{4})\\ +\epsilon(1-p_{4})(1-q_{3})\\ +\tau(1-p_{4})(1-q_{4})\end{array}\right)\end{array}\right)$},$$

$$\mathbf{v}=\mathbf{v}M.$$

$$\mathbf{v}{M}^{\prime }=0.$$

$$Adj(M^{\prime })M^{\prime }=0,$$

$$\begin{split}{\bm{v}}\cdot{\bm{f}}=&\left|\begin{array}[]{cccc}\tau p_{1}q_{1}+\epsilon p_{1}q_{2}+\epsilon p_{2}q_{1}+\xi p_{2}q_{2}-1&\mu p_{1}+\eta p_{2}-1&\mu q_{1}+\eta q_{2}-1&f_{1}\\ \epsilon p_{1}q_{3}+\xi p_{1}q_{4}+\tau p_{2}q_{3}+\epsilon p_{2}q_{4}&\eta p_{1}+\mu p_{2}-1&\mu q_{3}+\eta q_{4}&f_{2}\\ \epsilon p_{3}q_{1}+\tau p_{3}q_{2}+\xi p_{4}q_{1}+\epsilon p_{4}q_{2}&\mu p_{3}+\eta p_{4}&\eta q_{1}+\mu q_{2}-1&f_{3}\\ \xi p_{3}q_{3}+\epsilon p_{3}q_{4}+\epsilon p_{4}q_{3}+\tau p_{4}q_{4}&\eta p_{3}+\mu p_{4}&\eta q_{3}+\mu q_{4}&f_{4}\end{array}\right|\equiv D(\bm{p},\bm{q},\bm{f}),\end{split}$$

$$s_X=\frac{\mathbf{v}\cdot {\mathbf{S}}_X}{\mathbf{v}\cdot 1}=\frac{D(\mathbf{p},\mathbf{q},{\mathbf{S}}_X)}{D(\mathbf{p},\mathbf{q},1)},$$

$$s_Y=\frac{\mathbf{v}\cdot {\mathbf{S}}_Y}{\mathbf{v}\cdot 1}=\frac{D(\mathbf{p},\mathbf{q},{\mathbf{S}}_Y)}{D(\mathbf{p},\mathbf{q},1)},$$

$$\alpha s_X+\beta s_Y+\gamma =\frac{D(\mathbf{p},\mathbf{q},\alpha {\mathbf{S}}_{\mathbf{X}}+\beta {\mathbf{S}}_{\mathbf{Y}}+\gamma 1)}{D(\mathbf{p},\mathbf{q},1)},$$

$$\begin{split}&D(\bm{p},\bm{q},\alpha\bm{S}_{X}+\beta\bm{S}_{Y}+\gamma\bm{1})=\\ &\left|\begin{array}[]{cccc}\tau p_{1}q_{1}+\epsilon p_{1}q_{2}+\epsilon p_{2}q_{1}+\xi p_{2}q_{2}-1&\mu p_{1}+\eta p_{2}-1&\mu q_{1}+\eta q_{2}-1&\alpha R_{E}+\beta R_{E}+\gamma\\ \epsilon p_{1}q_{3}+\xi p_{1}q_{4}+\tau p_{2}q_{3}+\epsilon p_{2}q_{4}&\eta p_{1}+\mu p_{2}-1&\mu q_{3}+\eta q_{4}&\alpha S_{E}+\beta T_{E}+\gamma\\ \epsilon p_{3}q_{1}+\tau p_{3}q_{2}+\xi p_{4}q_{1}+\epsilon p_{4}q_{2}&\mu p_{3}+\eta p_{4}&\eta q_{1}+\mu q_{2}-1&\alpha T_{E}+\beta S_{E}+\gamma\\ \xi p_{3}q_{3}+\epsilon p_{3}q_{4}+\epsilon p_{4}q_{3}+\tau p_{4}q_{4}&\eta p_{3}+\mu p_{4}&\eta q_{3}+\mu q_{4}&\alpha P_{E}+\beta P_{E}+\gamma\end{array}\right|.\end{split}$$

$$\alpha s_X+\beta s_Y+\gamma =0.$$

$$det(\mathrm{A})=0$$

$$s{\mathbf{a}}_1+t{\mathbf{a}}_2+u{\mathbf{a}}_3+v{\mathbf{a}}_4=0,$$

$$D({\bm{p},\bm{q},\alpha\bm{S}_{X}+\beta\bm{S}_{Y}+\gamma\bm{1}})=\left|\begin{array}[]{cccc}p_{1}q_{1}-1&p_{1}-1&q_{1}-1&\alpha R+\beta R+\gamma\\ p_{2}q_{3}&p_{2}-1&q_{3}&\alpha S+\beta T+\gamma\\ p_{3}q_{2}&p_{3}&q_{2}-1&\alpha T+\beta S+\gamma\\ p_{4}q_{4}&p_{4}&q_{4}&\alpha P+\beta P+\gamma\end{array}\right|,$$

$$\begin{array}{rl}{p}_1-1& =\alpha R+\beta R+\gamma \\ p_2-1& =\alpha S+\beta T+\gamma \\ p_3& =\alpha T+\beta S+\gamma \\ p_4& =\alpha P+\beta P+\gamma ,\end{array}$$

$$p_1=p_2=p_3=p_4.$$

$$\alpha s_X+\beta s_Y+\gamma =0.$$

$$\begin{array}{rl}\mu p_1+\eta p_2-1& =\alpha R_E+\beta R_E+\gamma \\ \eta p_1+\mu p_2-1& =\alpha S_E+\beta T_E+\gamma \\ \mu p_3+\eta p_4& =\alpha T_E+\beta S_E+\gamma \\ \eta p_3+\mu p_4& =\alpha P_E+\beta P_E+\gamma ,\end{array}$$

$$p_1=p_2=p_3=p_4.$$

$$s\left(\begin{array}[]{c}p_{1}q_{1}-1\\ p_{2}q_{3}\\ p_{3}q_{2}\\ p_{4}q_{4}\end{array}\right)+t\left(\begin{array}[]{c}p_{1}-1\\ p_{2}-1\\ p_{3}\\ p_{4}\end{array}\right)+u\left(\begin{array}[]{c}q_{1}-1\\ q_{3}\\ q_{2}-1\\ q_{4}\end{array}\right)+v\left(\begin{array}[]{c}\alpha R+\beta R+\gamma\\ \alpha S+\beta T+\gamma\\ \alpha T+\beta S+\gamma\\ \alpha P+\beta P+\gamma\end{array}\right)=\bm{0}.$$

$$\left(\begin{array}[]{c}(sp_{1}+u)q_{1}\\ (sp_{2}+u)q_{3}\\ (sp_{3}+u)q_{2}\\ (sp_{4}+u)q_{4}\end{array}\right)+t\left(\begin{array}[]{c}p_{1}-1\\ p_{2}-1\\ p_{3}\\ p_{4}\end{array}\right)+\left(\begin{array}[]{c}-u-s\\ 0\\ -u\\ 0\end{array}\right)+v\left(\begin{array}[]{c}\alpha R+\beta R+\gamma\\ \alpha S+\beta T+\gamma\\ \alpha T+\beta S+\gamma\\ \alpha P+\beta P+\gamma\end{array}\right)=\bm{0}.$$

$$\{\begin{array}{ll}s{p}_1+u=0& \\ s{p}_2+u=0& \\ s{p}_3+u=0& \\ s{p}_4+u=0.& \end{array}$$

$$t\left(\begin{array}[]{c}p_{1}-1\\ p_{2}-1\\ p_{3}\\ p_{4}\end{array}\right)+\left(\begin{array}[]{c}-u-s\\ 0\\ -u\\ 0\end{array}\right)+v\left(\begin{array}[]{c}\alpha R+\beta R+\gamma\\ \alpha S+\beta T+\gamma\\ \alpha T+\beta S+\gamma\\ \alpha P+\beta P+\gamma\end{array}\right)=\bm{0}.$$

$$\{\begin{array}{ll}s(p_1-p_4)& =0\\ s(p_2-p_4)& =0\\ s(p_3-p_4)& =0\\ s{p}_4+u& =0.\end{array}$$

$$t\left(\begin{array}[]{c}p_{1}-1\\ p_{2}-1\\ p_{3}\\ p_{4}\end{array}\right)+v\left(\begin{array}[]{c}\alpha R+\beta R+\gamma\\ \alpha S+\beta T+\gamma\\ \alpha T+\beta S+\gamma\\ \alpha P+\beta P+\gamma\end{array}\right)=\bm{0}.$$

Full text

Strategies that enforce linear payoff relationships under observation errors in Repeated Prisoner’s Dilemma game

Azumi Mamiya1 and Genki Ichinose1∗  

 

1 Department of Mathematical and Systems Engineering, Shizuoka University,

3-5-1 Johoku, Naka-ku, Hamamatsu, 432-8561, Japan

∗ Corresponding author (ichinose.genki@shizuoka.ac.jp)

Abstract

The theory of repeated games analyzes the long-term relationship of interacting players and mathematically reveals the condition of how cooperation is achieved, which is not achieved in a one-shot game. In the repeated prisoner’s dilemma (RPD) game with no errors, zero-determinant (ZD) strategies allow a player to unilaterally set a linear relationship between the player’s own payoff and the opponent’s payoff regardless of the strategy that the opponent implements. In contrast, unconditional strategies such as ALLD and ALLC also unilaterally set a linear payoff relationship. Errors often happen between players in the real world. However, little is known about the existence of such strategies in the RPD game with errors. Here, we analytically search for all strategies that enforce a linear payoff relationship under observation errors in the RPD game. As a result, we found that, even in the case with observation errors, the only strategy sets that enforce a linear payoff relationship are either ZD strategies or unconditional strategies and that no other strategies can enforce it, which were numerically confirmed.

Keywords

Prisoner’s dilemma, repeated games, observation errors, zero-determinant strategies, unconditional strategies

1 Introduction

The two-player repeated prisoner’s dilemma (RPD) game is a model for exploring the long-term relationships of players, which mathematically reveals how cooperation and competition arise among competitive players [1]. In the one-shot PD game, defection is the only Nash equilibrium. On the other hand, cooperation is possible in the RPD game because players can reward cooperating partners by cooperating in the future. Also, players can punish defecting partners by defecting in the future. This mechanism is called direct reciprocity [2, 3, 4] and makes it possible for players to mutually cooperate in the RPD game. In the context of the RPD game, theoretical biologists are interested in which strategies win in evolving populations. This question falls into the field of evolutionary games [5]. A series of the results of evolutionary games in the RPD game brought promising findings. Especially, with noise, generous tit-for-tat [6] and win-stay lose-shift [7, 8] were robust to various kinds of evolutionary opponents. In this way, theoretical biologists have traditionally focused on strong strategies obtained from evolutionary consequences. However, we can ask a question from a different point of view: Are there any strategies which always win against the opponent irrespective of the opponent’s strategy? Answering this question fosters greater understanding of the RPD game.

In 2012, Press and Dyson suddenly answered this question by finding a novel class of strategies which contain such ultimate strategies, called zero-determinant (ZD) strategies [9]. ZD strategies impose a linear relationship between the payoffs for a focal player and his opponent regardless of the strategy that the opponent implements. The discovery of ZD strategies inspired various relevant studies, including their evolution [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25], multiplayer games [26, 19, 27, 28, 29], continuous action spaces [30, 28, 29, 31], alternating games [31], animal contests [32], human reactions to computerized ZD strategies [33, 34], and human-human experiments [35, 28, 36], which promote an understanding of the nature of human cooperation. For further understanding, see the recent elegant classification of strategies, partners (called “good strategies” in Ref. [11, 37]) and rivals, in direct reciprocity [38]. In contrast, unconditional strategies such as ALLC and ALLD can also unilaterally set a linear payoff relationship against the opponent [14, 39]. A previous study revealed that those two types of strategies are the only sets which enforce a linear payoff relationship in the RPD game [39].

These two types of strategies were found in the case of no errors. Errors (or noise) are unavoidable in human interactions and they may lead to the collapse of cooperation due to negative effects. Thus, the effect of errors has been considered in the literature of the RPD game [40, 41, 42, 43, 44, 45, 46, 47, 48]. However, except for [49], the effect of errors has not been considered for strategies that enforce a linear payoff relationship. There are typically two types of errors: perception errors [42] and implementation errors [43]. Hao et al. considered the former case of the errors where players may misunderstand their opponent’s action because the players can only rely on their private monitoring [40, 44] instead of their opponent’s direct action. They remarkably showed that ZD strategies can exist even in the case that such observation errors are incorporated [49]. In their model, they mathematically searched for one of the cases where determinants become zero in line with Press and Dyson’s formalism [9]. More specifically, they only searched for the case where the second and fourth columns of the determinant take the same value as Press and Dyson did in the case of no errors. They did not consider other possible strategies that make the determinant zero in the case of errors. In this study, from all possibilities, we mathematically searched for all of the cases where the determinant becomes zero. As a result, we found that only ZD strategies [9] and unconditional strategies [14, 39] are the two types which enforce a linear payoff relationship and that no other strategies exist to make the determinant zero. We also confirmed this result by numerical calculations.

2 Model

We consider the symmetric two-person RPD game with observation errors in line with the previous studies [44, 49]. Each player $i\in\{X,Y\}$ chooses an action $a_{i}\in\{\rm C,\rm D\}$. Each player cannot see what action the opponent chose. Instead, they can only observe a signal $\omega_{i}\in\{g,b\}$, where $g$ and $b$ denote good and bad signals, respectively. The signal cannot be observed by the other player, meaning that the signal is private information. Each player’s signal $\omega_{i}$ basically depends on the opponent’s action but is also affected by noise from the environment, which is a stochastic variable. In other words, a player observes $g$ (or $b$) when the other player chooses an action C (or D). However, when an error occurs, a player observes $b$ (or $g$) although the other player chooses an action C (or D) due to observation errors. We define $\sigma(\omega|a)$ as the probability that a signal profile $\omega=(\omega_{X},\omega_{Y})$ is realized, given that an action profile $a=(a_{X},a_{Y})$ occurs. Let $\epsilon$ be the probability that an error happens to one particular player but not to the other and $\xi$ be the probability that an error happens to both players. Then, the probability that an error occurs to neither player is $1-2\epsilon-\xi$. For example, when both players choose action C, we have $\sigma(g,g|\rm C,\rm C)=1-2\epsilon-\xi$, $\sigma(b,g|\rm C,\rm C)=\sigma(\it{g,b}|\rm C,\rm C)=\epsilon$, and $\sigma(b,b|\rm C,\rm C)=\xi$. The realized payoff for each player depends only on the action he chose and the signal he received, which is denoted by $u_{i}(a_{i},\omega_{i})$. Let $u_{i}(\rm{C},\it{g})$, $u_{i}(\rm{C},\it{b})$, $u_{i}(\rm{D},\it{g})$, and $u_{i}(\rm{D},\it{b})$ be $R,S,T$, and $P$, respectively. Then the payoff matrix is given by

[TABLE]

The entries represent the payoffs that a focal player gains in a single round of the repeated game. Each row and column represents the action that the focal player chose and the signal he observed, respectively. In each stage, player $i$’s expected payoff value over all possible signals, when two players have an action profile $a$, is represented by

[TABLE]

The expected payoffs under different action profiles $(\rm C,\rm C)$,$(\rm C,\rm D)$,$(\rm D,\rm C)$, and $(\rm D,\rm D)$ are denoted by $R_{E}$, $S_{E}$, $T_{E}$ and $P_{E}$, respectively. According to Eq. (2), $R_{E}$, $S_{E}$, $T_{E}$, and $P_{E}$ are derived as $R_{E}=R(1-\epsilon-\xi)+S(\epsilon+\xi)$, $S_{E}=S(1-\epsilon-\xi)+R(\epsilon+\xi)$, $T_{E}=T(1-\epsilon-\xi)+P(\epsilon+\xi)$, $P_{E}=P(1-\epsilon-\xi)+T(\epsilon+\xi)$, respectively. We assume that

[TABLE]

which dictates the prisoner’s dilemma condition. Both players expect a larger payoff by selecting D rather than C irrespective of the other’s action because $T_{E}>R_{E}$ and $P_{E}>S_{E}$ hold. We also assume that

[TABLE]

which guarantees that mutual cooperation is more beneficial than the two players alternating C and D in the opposite phase, i.e., CD, DC, CD, DC, $\ldots$, where the first and second letter represent the actions selected by $X$ and $Y$, respectively. The two players repeat the game whose payoff matrix in each round is given by Eq. (1).

Consider two players $X$ and $Y$ that adopt memory-one strategies, with which they use only the outcomes of the last round to decide the action to be submitted in the current round. Even in the case of memory-$n$ strategies, errors can be considered. In fact, Hilbe et al. incorporated implementation errors in such a situation [48]. A memory-one strategy is specified by a 4-tuple; $X$’s strategy is given by a combination of

[TABLE]

where $0\leq p_{\rm 1},p_{\rm 2},p_{\rm 3},p_{\rm 4}\leq 1$. The subscripts 1, 2, 3, and 4 of $p$ mean previous outcome C$g$, C$b$, D$g$ and D$b$, respectively. In Eq. (5), $p_{\rm 1}$ is the conditional probability that $X$ cooperates when $X$ cooperated and observed signal $g$ in the last round, $p_{\rm 2}$ is the conditional probability that $X$ cooperates when $X$ cooperated and observed signal $b$ in the last round, $p_{\rm 3}$ is the conditional probability that $X$ cooperates when $X$ defected and observed signal $g$ in the last round, and $p_{\rm 4}$ is the conditional probability that $X$ cooperates when $X$ defected and observed signal $b$ in the last round. Note that, in this model, $\bm{p}$ depends on $X$’s action and its private observation in the last round [44, 49]. Contrary, $\bm{p}$ depends on $X$’s and $Y$’s direct actions in the last round in the case of no errors. Similarly, $Y$’s strategy is specified by a combination of

[TABLE]

where $0\leq q_{\rm 1},q_{\rm 2},q_{\rm 3},q_{\rm 4}\leq 1$. Because both players adopt a memory-one strategy, the stochastic state of the two players in round $t$ is described by $\bm{v}(t)=(v_{1}(t),v_{2}(t),v_{3}(t),v_{4}(t))$, where the subscripts 1, 2, 3, and 4 of $v$ mean the stochastic state (C,C), (C,D), (D,C), and (D,D), respectively. $v_{1}(t)$ is the probability that both players cooperate in round $t$, $v_{2}(t)$ is the probability that $X$ cooperates and $Y$ defects in round $t$, and so forth. The state transition matrix $M$ of this noisy repeated game is given by

[TABLE]

where $\tau=1-2\epsilon-\xi$. Each row and column represents the previous states and the following states of the game, respectively. Then, the stochastic state of the two players in round $t+1$ is calculated by $\bm{v}(t+1)=\bm{v}(t)M$. The stationary distribution for $M$ is a vector $\bm{v}$ such that

[TABLE]

Eq. (8) and $M^{\prime}\equiv M-I$ yield

[TABLE]

Applying Cramer’s rule to matrix $M^{\prime}$, we obtain

[TABLE]

where $Adj(M^{\prime})$ is the adjugate matrix of $M^{\prime}$. Here, Eqs. (9) and (10) imply that every row of $Adj(M^{\prime})$ is proportional to $\bm{v}$. Therefore, $\bm{v}$ is solely represented by the components of matrix $M^{\prime}$. Choosing the fourth row of the matrix $Adj(M^{\prime})$, we see that $\bm{v}$ is composed of the determinant of the $3\times 3$ matrixes formed from the first three columns of $M^{\prime}$. We add the first column of $M^{\prime}$ into the second and third columns. Even by this manipulation, this determinant is unchanged. The result of these manipulations is a formula for the dot product of an arbitrary vector $\bm{f}=(f_{1},f_{2},f_{3},f_{4})$ with the stationary distribution vector $\bm{v}$, which can be represented by the form of the determinant

[TABLE]

where $\mu=1-\epsilon-\xi$ and $\eta=\epsilon+\xi$. If we replace the arbitrary vector $\bm{f}$ with $X$’s expected payoff vector $\bm{S}_{X}=(R_{E},S_{E},T_{E},P_{E})$, we obtain $\bm{v}\cdot\bm{S}_{X}$. Then, we divide it by $\bm{v}\cdot\bm{1}$. Finally, we can obtain player $X$’s per-round expected payoff in the form of the determinant as follows:

[TABLE]

where $\bm{1}=(1,1,1,1)$ is needed for the normalization. Similarly, player $Y$’s per-round payoff can be represented by the form of the determinant

[TABLE]

where $\bm{S}_{Y}$ is $Y$’s expected payoff vector $(R_{E},T_{E},S_{E},P_{E})$. Hereafter, we only consider the relationship between those two expected payoffs because they converge to certain expected values, respectively, if the stationary distributions exist in infinitely repeated games. In contrast, other types of the payoff are worth investigating in finitely repeated games.

Moreover, we can consider the linear combination of $s_{X}$ and $s_{Y}$, which can be given by the form of the determinant

[TABLE]

where $\alpha,\beta$, and $\gamma$, are arbitrary constant. The numerator of the right side of Eq. (14) is expressed in the following:

[TABLE]

If Eq. (15) is zero, the relationship between the two players’ payoffs becomes linear. In the next section, we search for all of the solutions which satisfy this condition.

3 Result

We search for strategies that impose a linear relationship between the two players’ payoffs regardless of their opponent’s strategies in the RPD game with observation errors, which satisfy the following equation:

[TABLE]

If the numerator of the right side of Eq. (14) is zero, Eq. (16) holds. In other words, if $D(\bm{p},\bm{q},\alpha\bm{S}_{X}+\beta\bm{S}_{Y}+\gamma\bm{1})=0$ is satisfied, there is a linear payoff relationship between the two players’ payoffs.

Press and Dyson (without error) [9] and Hao et al. (with error) [49] only searched for the case that the second and fourth columns take the same value. This makes the determinant become zero. Here, from all possibilities, we search for all of the cases (including this case) that $D(\bm{p},\bm{q},\alpha\bm{S}_{X}+\beta\bm{S}_{Y}+\gamma\bm{1})=0$ holds. The following determinant theorem gives such a condition.

Theorem 1

For an $n\times n$ matrix A, the following holds:

[TABLE]

We define $\bm{a}_{i}$ ($i\in\{1,2,3,4\}$) as $i$-th column vector of the determinant of Eq. (15). From the above theorem, if the columns of the determinant of Eq. (15) are linearly dependent vectors, there exist real numbers $s,t,u,v,\alpha,\beta$, and $\gamma$, except for the trivial solution ($(s,t,u,v)=(0,0,0,0)$,$(\alpha,\beta,\gamma)=(0,0,0)$), such that

[TABLE]

where vector $\bm{0}$ denotes a zero vector.

3.1 Without errors (perfect monitoring)

3.1.1 Mathematical analysis

In this section, we search for all of the strategies that enforce a linear payoff relationship without errors ($\epsilon=0$ and $\xi=0$). When there are no errors, the expected payoffs correspond to the original payoffs, i.e., $\bm{S}_{X}=(R_{E},S_{E},T_{E},P_{E})=(R,S,T,P)$ and $\bm{S}_{Y}=(R_{E},T_{E},S_{E},P_{E})=(R,T,S,P)$, respectively. In addition, by substituting $\epsilon=0$ and $\xi=0$ into Eq. (15), we obtain

[TABLE]

which is the same with Press and Dyson’s determinant [9]. By the extensive calculations provided in Appendix A, we found the only strategies that impose a linear payoff relationship between the two players’ payoffs are either

[TABLE]

or

[TABLE]

Equation (19) corresponds to ZD strategies without error ([9], Eq. (1) of [14], Eq. (1) of [13], and Eq. (3) of [38]). Equation (20) is called unconditional strategies [14]. Only these strategy sets $\bm{p}$ can impose a linear relationship and no other strategies can impose it.

To conclude, in the RPD game under perfect monitoring, we showed that either ZD strategies or unconditional strategies can impose a linear relationship between the two players’ payoffs. This is consistent with the previous result in the case with a discount factor but no errors [39].

3.1.2 Numerical examples

We show numerical examples that ZD strategies and unconditional strategies can impose a linear relationship between the two players’ payoffs while others cannot in the RPD game without errors. Figure 1 shows the relationship between the two players’ expected payoffs per game with payoff vector $(T,R,P,S)=(1.5,1,0,-0.5)$. The gray quadrangle in each panel represents the feasible set of the payoffs. We fixed one particular strategy for player $X$ (vertical line) and randomly generate 1,000 strategies that satisfy $0\leq q_{1},q_{2},q_{3},q_{4}\leq 1$ for player $Y$ (horizontal axis). Thus, each black dot represents the payoff relationship between two players. In addition, the blue and red are the particular cases for player $Y$. Red is the case that player $Y$ is ALLD and blue is the case that player $Y$ is ALLC.

Figure 1A shows the case with a Win-Stay-Lose-Shift (WSLS) strategy vs. $1000+2$ strategies. As WSLS strategies are neither ZD nor unconditional strategies, the payoff relationships are not linear.

Numerical examples of ZD strategies

Equalizer [9], Extortioner [9], and Generous strategies [23] are known as the three most prominent ZD strategies. Here, we take up the first two as the numerical examples of ZD although Generous strategies play an important role in the evolution of cooperation. In contrast to Extortion, Generous strategies always obtain lower payoffs than the opponent except for mutual cooperation. Hence, Generous strategies are known as one of the cooperative ZD strategies. Because Extortion never loses in a one-to-one competition, Extortion is feasible in a small population. However, in a large evolving population, cooperative groups are more successful than the group of Extortioners. Thus, evolution leads from Extortion to Generous strategies [23]. In this sense, Generous strategies are important. Figure 1B is the case with an Extortioner strategy vs. $1000+2$ strategies. Extortioner strategies are the subset of ZD strategies [9] (See Box 1 in [38] for a clear explanation of Extortioner (extortionate) strategies). Extortioner strategies can always gain a higher payoff than the one’s opponent, except for the point $(P,P)$, regardless of the opponent’s strategies. When we set $(\alpha,\beta,\gamma)=(0.01,-0.15,0)$ in Eq. (19), we obtain an Extortioner strategy, $\bm{p}=(0.86,0.77,0.09,0)$, with $0.01s_{X}-0.15s_{Y}=0$. In this particular case, the Extortioner strategy (player $X$) gains the payoff fifteen times higher than player $Y$.

Figure 1C is the case with an Equalizer strategy vs. $1000+2$ strategies. Note that, only in this case, the vertical and horizontal axes are reversed. Thus, the horizontal axis is the payoff of Equalizer (player $X$) and the vertical axis is the payoff of player $Y$. Equalizer strategies are also the subset of ZD strategies [9]. If a player uses Equalizer strategies, he can fix the opponent’s payoff to be one particular value. When we set $(\alpha,\beta,\gamma)=(0,-2/3,1/3)$ in Eq. (19), we obtain an Equalizer strategy, $\bm{p}=(2/3,1/3,2/3,1/3)$, which can fix the opponent’s payoff at $s_{Y}=0.5$ irrespective of the opponent’s strategies.

Figure 1D is the case with TFT $\bm{p}=(1,0,1,0)$ strategy vs. $1000+2$ strategies. When we set $(\alpha,\beta,\gamma)=(0.5,-0.5,0)$ in Eq. (19), we obtain TFT $\bm{p}=(1,0,1,0)$, which means that TFT is also the subset of ZD strategies. Actually, TFT is a special case of ZD strategies with $s_{X}=s_{Y}$ called “fair strategies” [26]. Moreover, the strategies that $p_{1}=1,p_{4}=0,p_{2}+p_{3}=1$ including TFT can impose the linear payoff relationship $s_{X}=s_{Y}$. See Appendix B for the proof.

Numerical examples of unconditional strategies

Figure 1E is the case with ALLC vs. $1000+2$ strategies. ALLC is one of the examples of unconditional strategies $(r,r,r,r),0\leq r\leq 1$ where $r=1$. If we substitute $r=1$ and $(T,R,P,S)=(1.5,1,0,-0.5)$ into Eq. (34), we obtain $(\beta,\gamma)=(3\alpha,-4\alpha)$ and we have a straight line represented by $s_{X}+3s_{Y}-4=0$. We numerically see that the payoff of ALLC is always lower than the opponent’s payoff except for $(R,R)$.

Figure 1F is the case with ALLD vs. $1000+2$ strategies. ALLD is also one of the examples of unconditional strategies $(r,r,r,r),0\leq r\leq 1$ where $r=0$. If we substitute $r=0$ and $(T,R,P,S)=(1.5,1,0,-0.5)$ into Eq. (34), we obtain $(\beta,\gamma)=(3\alpha,0)$ and we have a straight line represented by $s_{X}+3s_{Y}=0$. We numerically see that the payoff of ALLD is always higher than the opponent’s payoff except for $(P,P)$. Unlike ZD strategies, the slopes of the straight lines in Figure 1E and 1F are always negative [14].

3.2 With observation errors (imperfect monitoring)

3.2.1 Mathematical analysis

In the same way as no errors, we search for strategies that impose a linear relationship between the two players’ payoffs regardless of the opponent’s strategy in the RPD game with observation errors. If the numerator of the right side of Eq. (14) is zero, the following equation holds:

[TABLE]

In other words, if $D(\bm{p},\bm{q},\alpha\bm{S}_{X}+\beta\bm{S}_{Y}+\gamma\bm{1})=0$ is satisfied, there is a linear payoff relationship between the two players’ payoffs. By the extensive calculations provided in Appendix C, we found the only strategies that impose a linear payoff relationship between the two players’ payoffs are either

[TABLE]

or

[TABLE]

Equation (22) is ZD strategies with observation errors. This is consistent with Hao et al.’s [49]. Equation (23) is unconditional strategies. Moreover, we analytically show the feasible payoff range for unconditional strategies. See Appendix D.

In summary, in the RPD game even with observation errors (imperfect monitoring), we showed that either ZD strategies or unconditional strategies can impose a linear relationship between the two players’ payoffs and that no other strategies can impose it. This is a new fact discovered in this study.

3.2.2 Numerical examples

As well as the case without errors, we show numerical examples that ZD strategies and unconditional strategies can impose a linear relationship between the two players’ payoffs while others cannot in the RPD game with errors. Figure 2 shows the relationship between the two players’ expected payoffs per game with payoff vector $(T,R,P,S)=(1.5,1,0,-0.5)$. The gray quadrangle in each panel represents the feasible payoff set. As error rates are increased, the size of the feasible payoff set becomes smaller. We fixed one particular strategy for player $X$ (vertical line) and randomly generate 1,000 strategies that satisfy $0\leq q_{1},q_{2},q_{3},q_{4}\leq 1$ for player $Y$ (horizontal axis). Each black dot represents the payoff relationship between two players without errors ($\epsilon+\xi=0$), the same as Figure 1. Moreover, green, light green, and light blue dots correspond to the cases of $\epsilon+\xi=0.1,0.2$, and 0.3, respectively. We do not consider the case of $\epsilon+\xi\geq 1/3$ because it does not satisfy the prisoner’s dilemma condition: $T_{E}>R_{E}>P_{E}>S_{E}$. As in the case with no errors, red is the case that player $Y$ is ALLD and blue is the case that player $Y$ is ALLC.

Figure 2A shows the case with a Win-Stay-Lose-Shift (WSLS) strategy vs. $1000+2$ strategies. In this case, $\xi=0$ is fixed and $\epsilon$ is varied to 0.1, 0.2 and 0.3. As in the case with no errors, the payoff relationships are not linear in this case because WSLS strategies are neither ZD nor unconditional strategies.

Numerical examples of ZD strategies

Figure 2B is the case with an Extortioner strategy vs. $1000+2$ strategies. As shown in Figure 1, $\bm{p}=(0.86,0.77,0.09,0)$ (black dots) is the extortion strategy without errors. In this case, player $X$ can always gain a higher payoff than the opponent (with the slope of 15), except for the point $(P,P)$, regardless of the opponent’s strategies. $\bm{p}=(0.926875,0.818125,0.111875,0.003125)$ (green) and $\bm{p}=(1,0.86,0.14,0)$ (light green) are the extortion strategies when $\epsilon+\xi=0.1$ and 0.2, respectively. Unlike Extortioner without errors, there exists the region that the expected payoff of the Extortioner with errors is lower than the opponent’s payoff near $(P_{E},P_{E})$.

Hao et al. already proved this fact [49]. They call it dominant extortion when the expected payoff of a focal player is always higher than the opponent except for $(P,P)$. This is only possible when there are no errors. When there are errors, only contingent extortion can exist as Hao et al. proved. We assume that player $X$ adopts the contingent extortion. The contingent extortion implies that when player $Y$ tries to increase his payoff, he will increase $X$’s payoff even more. However, in some regions near $(P_{E},P_{E})$, $X$’s payoff is lower than $Y$’s payoff. We mathematically restate the difference between dominant and contingent based on Hao et al.’s formalism [49]. We transform $\alpha=\phi s^{\prime},\beta=-\phi,\gamma=\phi(1-s^{\prime})l$ in Eq. (22) in line with Hilbe’s formalism [14]. We determine $l,s^{\prime}$ so that $l=P_{E}+\Delta$，$1/s^{\prime}>1$ are satisfied where $1/s^{\prime}$ is the slope of the line. Note that the inverse of $s^{\prime}$ is considered as the slope because, in Hilbe’s formalism, $s^{\prime}$ is the coefficient for player $Y$ while in our and Hao’s formalism $s^{\prime}$ is the coefficient for player $X$. Also, $\phi,\Delta$ must satisfy $0\leq p_{1},p_{2},p_{3},p_{4}\leq 1$. When $\epsilon+\xi=0$ (no error), if we set $(s^{\prime},\phi,\Delta)=(1/15,0.15,0)$, we obtain $l=P_{E}$ in Eq. (22) and $\bm{p}$ becomes $\bm{p}=(0.86,0.77,0.09,0)$ (black dots in Figure 2B). In this case, the payoff of player $X$ is always higher than player $Y$ except for the point $(P,P)$. However, when $\epsilon+\xi>0$, there is no solution in Eq. (22) when $\Delta=0$. Thus, $\Delta>0$ is needed, which means that there are the cases that the payoff of player $X$ is lower than that of player $Y$. For instance, when $\epsilon+\xi=0.1,0.2$ are given, if we set $(s^{\prime},\phi,\Delta)=(1/15,0.15,0.1)$ and $(s^{\prime},\phi,\Delta)=(1/15,0.15,0.2)$, $\bm{p}=(0.926875,0.818125,0.111875,0.003125)$ (green in Figure 2B) $\bm{p}=(1,0.86,0.14,0)$ (light green in Figure 2B) are obtained. In those cases, $X$’s payoff is lower than $Y$’s payoff near $(P_{E},P_{E})$ although $Y$’s increase leads to $X$’s increase even more.

Figure 2C is the case with an Equalizer strategy vs. $1000+2$ strategies. Note that, only in this case, the vertical and horizontal axes are reversed. Thus, the horizontal axis is the payoff of Equalizer (player $X$) and the vertical axis is the payoff of player $Y$. As Hao et al. already suggested [49], there exist Equalizer strategies even if errors are incorporated. When $\epsilon+\xi=0$ (no error), if we set $(\alpha,\beta,\gamma)=(0,-2/3,1/3)$ in Eq. (22), we obtain an Equalizer strategy, $\bm{p}=(2/3,1/3,2/3,1/3)$, which can fix the opponent payoff at $s_{Y}=0.5$ irrespective of the opponent’s strategies as shown by black dots in Figure 2C. When $\epsilon+\xi=0.1$, if we set $(\alpha,\beta,\gamma)=(0,-0.695653,0.347827)$, we obtain an Equalizer strategy, $\bm{p}=(0.8,0.365217,0.634783,0.2)$ which can fix the opponent payoff at $s_{Y}=0.5$ as shown by green dots in Figure 2C. Also, when $\epsilon+\xi=0.2$, if we set $(\alpha,\beta,\gamma)=(0,-0.3,0.15)$, we obtain an Equalizer strategy, $\bm{p}=(0.99,0.74,0.26,0.01)$ which can fix the opponent payoff at $s_{Y}=0.5$ as shown by light green dots in Figure 2C. As error rates are increased, the payoff range for Equalizer becomes smaller.

Figure 2D is the case with TFT $\bm{p}=(1,0,1,0)$ strategy vs. $1000+2$ strategies. When $\epsilon+\xi=0$ (no error), if we set $\bm{p}=(1,0,1,0)$ in Eq. (22), we obtain $(\alpha,\beta,\gamma)=(0.5,-0.5,0)$, which means that $s_{X}=s_{Y}$ (black dots) in the case of TFT. When $\epsilon+\xi=0.1,0.2$, and 0.3, if we set $\bm{p}=(1,0,1,0)$ in Eq. (22), we obtain $(\alpha,\beta,\gamma)=(0.386555,-0.672269,0.142857)$, $(\alpha,\beta,\gamma)=(0.0714286,-1.07143,0.5)$, and $(\alpha,\beta,\gamma)=(-2.36364,-3.63636,3)$, respectively. Thus, we obtain the corresponding lines, $(\alpha,\beta,\gamma)=(0.386555,-0.672269,0.142857)$ (green), $(\alpha,\beta,\gamma)=(0.0714286,-1.07143,0.5)$ (light green), and $(\alpha,\beta,\gamma)=(-2.36364,-3.63636,3)$ (light blue), respectively. When there are no errors, $s_{X}=s_{Y}$ always holds. However, there are errors, this does not hold any more. As error rates are increased, the difference between $s_{X}$ and $s_{Y}$ becomes larger. In general, when there are errors, unlike when there are no errors, TFT does not enforce a linear payoff relationship. Only when special payoff matrices are given, the linear payoff relationship remains. See Appendix E in detail.

Numerical examples of unconditional strategies

Figure 2E is the case with ALLC vs. $1000+2$ strategies. ALLC is one of the examples of unconditional strategies $(r,r,r,r),0\leq r\leq 1$ where $r=1$. When $\epsilon+\xi=0$ (no error), by Eq. (51), we obtain $(\beta,\gamma)=(3\alpha,-4\alpha)$. Thus, the equation of the straight line is $s_{X}+3s_{Y}-4=0$ (black dots in Figure 2E) and the domain of $s_{X}$ becomes $-0.5\leq s_{X}\leq 1$ from Eq. (66). When $\epsilon+\xi=0.1,0.2$, and 0.3, we obtain the corresponding lines, $s_{X}+2.4s_{Y}-2.89=0\,(-0.35\leq s_{X}\leq 0.85)$ (green)，$s_{X}+1.8s_{Y}-1.96=0\,(-0.2\leq s_{X}\leq 0.7)$ (light green)，and $s_{X}+1.2s_{Y}-1.21=0\,(-0.05\leq s_{X}\leq 0.55)$ (light blue), respectively. We numerically see that the payoff of ALLC is always lower than the opponent’s payoff except for $(R_{E},R_{E})$ and all the dots are on the feasible lines $(R_{E},R_{E})-(T_{E},S_{E})$, respectively.

Figure 2F is the case with ALLD vs. $1000+2$ strategies. ALLD is also one of the examples of unconditional strategies $(r,r,r,r),0\leq r\leq 1$ where $r=0$. When $\epsilon+\xi=0$ (no error), by Eq. (51), we obtain $(\beta,\gamma)=(3\alpha,0)$. Thus, the equation of the straight line is $s_{X}+3s_{Y}=0$ (black dots in Figure 2F) and the domain of $s_{X}$ becomes $0\leq s_{X}\leq 1.5$ from Eq. (66). When $\epsilon+\xi=0.1,0.2$, and 0.3, we obtain the corresponding lines, $s_{X}+2.4s_{Y}-0.51=0\,(0.15\leq s_{X}\leq 1.35)$ (green)，$s_{X}+1.8s_{Y}-0.84=0\,(0.3\leq s_{X}\leq 1.2)$ (light green)，and $s_{X}+1.2s_{Y}-0.99=0\,(0.45\leq s_{X}\leq 1.05)$ (light blue), respectively. We numerically see that the payoff of ALLD is always higher than the opponent’s payoff except for $(P_{E},P_{E})$ and all the dots are on the feasible lines $(S_{E},T_{E})-(P_{E},P_{E})$, respectively.

4 Conclusions

We analyzed strategies that enforce linear payoff relationships under observation errors in the RPD game. Press and Dyson firstly developed a new mathematical formalism for the expected payoffs of two players and found that if the second and fourth columns of the specific determinant take the same value, the determinant becomes zero, which implies the two players’ expected payoffs become linear [9]. Hao et al. used the same linear algebra technique and extended it to the case with observation errors [49]. Here, not just the case where the second and fourth columns of the determinant take the same value, we searched for all of the strategies which make the determinant zero under observation errors. As a result, we found that the only strategy sets that enforce a linear payoff relationship are either ZD strategies or unconditional strategies, which was consistent with the case of the RPD game with a discount factor [39]. We confirmed that the solutions are correct by showing some numerical calculations.

Press and Dyson first discovered strategies that make the determinant for the expected payoffs zero by finding that the second and fourth columns of the determinant take the same value [9]. They call these strategies “zero-determinant strategies” (original ZD strategies) and all subsequent studies also call them “zero-determinant strategies.” By searching for all possibilities, we found that not only these original ZD strategies but also unconditional strategies make the determinant zero with a different form and that no other strategies exist to make the determinant zero. In this sense, strictly speaking, both the original ZD strategies and unconditional strategies may be called “zero-determinant strategies.”

The original ZD strategies and the unconditional strategies are the only sets which impose a linear payoff relationship irrespective of the opponent strategies, not only in the case with a discount factor [39] but also in the case with observation errors as shown here. This result suggests that, in any case, those two sets are the only types of strategies that enforce a linear payoff relationship between two players. To investigate the inference, one possible direction of future research is analyzing the case of the RPD game with a discount factor under observation errors.

Appendix A Detailed calculations without errors

We substitute the column vectors of the determinant of Eq. (18) into Eq. (17) to obtain

[TABLE]

By taking out $\bm{q}$ in Eq. (24), we obtain

[TABLE]

Here, we search for strategies which satisfy $D(\bm{p},\bm{q},\alpha\bm{S}_{X}+\beta\bm{S}_{Y}+\gamma\bm{1})=0$ irrespective of $Y$’s strategy $\bm{q}$, meaning that Eq. (25) must hold true irrespective of $\bm{q}$. Therefore, the coefficients of each element $\bm{q}$ in Eq. (25) must equal to zero, that is, the following conditions are necessary:

[TABLE]

When Eq. (26) holds, the first terms of Eq. (25) are eliminated and we obtain

[TABLE]

If there exist real numbers, $s,t,u,v,\alpha,\beta$, and $\gamma$ such that Eq. (26) and Eq. (27) are satisfied simultaneously, $D(\bm{p},\bm{q},\alpha\bm{S}_{X}+\beta\bm{S}_{Y}+\gamma\bm{1})=0$ holds irrespective of $\bm{q}$. To solve Eq. (26), we subtract the fourth equation from the first three in Eq. (26):

[TABLE]

Then, we obtain $s=0$ or $p_{1}=p_{4}$ from the first equation. First, in the case that $s=0$ holds, the second and third equations automatically hold and we obtain $u=0$ from the fourth. Hence, we obtain $s=0$ and $u=0$. Second, in the cases that $s\neq 0$ and $p_{1}=p_{4}$ hold, we obtain $p_{2}=p_{4}$ and $p_{3}=p_{4}$ and $p_{4}=-u/s$ from the second, third and fourth equations, respectively. Therefore, the solutions of Eq. (26) are either (1) $s=0$ and $u=0$ or (2) $p_{1}=p_{2}=p_{3}=p_{4}=-u/s$. Next, we check that these solutions can also satisfy Eq. (27) in the following.

Case (1) $s=0$ and $u=0$:

In this case, we substitute $s=0$ and $u=0$ into Eq. (27) to obtain

[TABLE]

Here, when we set $t=0$, either equation

[TABLE]

or

[TABLE]

must hold. When we set $v=0$, we obtain the trivial solution $(s,t,u,v)=(0,0,0,0)$. Also, we solve Eq. (31) and obtain the trivial solution $(\alpha,\beta,\gamma)=(0,0,0)$. Hence, we do not have to consider the case of $t=0$. Therefore, in the following, we only consider $t\neq 0$. Replacing constants $-\alpha v/t$, $-\beta v/t$, and $-\gamma v/t$ with $\alpha$, $\beta$, and $\gamma$, we obtain,

[TABLE]

If there exist $\alpha,\beta$, and $\gamma$ for $\bm{p}$ satisfying Eq. (32), there must be solutions that Eq. (17) hold. This strategy set $\bm{p}$ can impose a linear relationship. Eq. (32) corresponds to ZD strategies without error ([9], Eq. (1) of [14], Eq. (1) of [13], and Eq. (3) of [38]).

Case (2) $p_{1}=p_{2}=p_{3}=p_{4}=-u/s$:

In this case, let $r\ (0\leq r\leq 1)$ be $-u/s$ , we substitute $p_{1}=p_{2}=p_{3}=p_{4}=r$ and $u=-sr$ into Eq. (27) to obtain

[TABLE]

There exist real numbers $s,t,u,v,\alpha,\beta$, and $\gamma$ which satisfies Eq. (33) as follows:

[TABLE]

Because there exist real numbers $s,t,u,v,\alpha,\beta$, and $\gamma$ such that Eq. (26) and Eq. (27) are satisfied, $p_{1}=p_{2}=p_{3}=p_{4}=r\ (0\leq r\leq 1)$ enforces a linear payoff relationship. This strategy set is called unconditional strategies [14]. By transforming $\alpha,\beta,\gamma$ into $\alpha=\phi s^{\prime},\beta=-\phi,\gamma=\phi(1-s^{\prime})l$ in Eq. (34), we obtain the following equations, which are the same as Eq. (16) of [14]:

[TABLE]

Appendix B Strategies that enforce $s_{X}=s_{Y}$ without errors

We prove that strategies specified by $p_{1}=1,p_{4}=0$ and $p_{2}+p_{3}=1$ including TFT enforce a linear payoff relationship with $s_{X}=s_{Y}$ under no errors. Equation (19) can be rewritten as follows by transforming $\alpha,\beta,\gamma$ into $\alpha=\phi s^{\prime},\beta=-\phi,\gamma=\phi(1-s^{\prime})l$ where $s^{\prime}$ is the slope of the straight line:

[TABLE]

which corresponds to Eq. (16) of [14]. When $s^{\prime}=1$, we obtain

[TABLE]

This gives $(\alpha,\beta,\gamma)=(\phi,-\phi,0)$, hence, we obtain $s_{X}=s_{Y}$. Thus, strategies specified by $p_{1}=1,p_{4}=0$ and $p_{2}+p_{3}=1$ enforce a linear payoff relationship with $s_{X}=s_{Y}$.

Appendix C Detailed calculations with errors

We substitute the column vectors of the determinant of Eq. (15) into Eq. (17) to obtain

[TABLE]

By taking out $\bm{q}$ in Eq. (38), we obtain

[TABLE]

Here, we search for strategies which satisfy $D(\bm{p},\bm{q},\alpha\bm{S}_{X}+\beta\bm{S}_{Y}+\gamma\bm{1})=0$ irrespective of $Y$’s strategy $\bm{q}$, meaning that Eq. (39) must hold true irrespective of $\bm{q}$. Therefore, the coefficients of each element $\bm{q}$ in Eq. (39) must equal to zero, that is, the following conditions are necessary:

[TABLE]

When Eq. (40) holds, the first terms of Eq. (39) are eliminated and we obtain

[TABLE]

If there exist real numbers, $s,t,u,v,\alpha,\beta$, and $\gamma$ such that Eq. (40) and Eq. (41) are satisfied simultaneously, $D(\bm{p},\bm{q},\alpha\bm{S}_{X}+\beta\bm{S}_{Y}+\gamma\bm{1})=0$ holds irrespective of $\bm{q}$. To solve Eq. (40), we subtract the sixth equation from the first, the seventh from the second, the fifth from the third, and the eighth from the fourth in Eq. (40) to obtain:

[TABLE]

First, we solve the first four equations and obtain (1) $s=0$, (2) $\epsilon-\xi=0$ and $1-3\epsilon-\xi=0$, (3) $p_{1}-p_{2}=0$ and $p_{3}-p_{4}=0$. We further analyze whether these equations satisfy the last four equations and Eq. (41) by dividing into three cases as follows.

Case (1) $s=0$:

In this case, we substitute $s=0$ into Eq. (42) to obtain

[TABLE]

where $\mu=1-\epsilon-\xi$ and $\eta=\epsilon+\xi$. The equations $\mu=0$ and $\eta=0$ do not hold at the same time. Therefore one of the solutions of Eq. (42) is $s=0$ and $u=0$. Next, we check whether this solution satisfies Eq. (41). We substitute $s=0$ and $u=0$ into Eq. (41) to obtain

[TABLE]

Here, when we set $t=0$, either equation

[TABLE]

or

[TABLE]

must hold. When we set $v=0$, we obtain the trivial solution $(s,t,u,v)=(0,0,0,0)$. Also, we solve Eq. (46) and obtain the trivial solution $(\alpha,\beta,\gamma)=(0,0,0)$. Hence, we do not have to consider the case of $t=0$. Therefore, in the following, we only consider $t\neq 0$. Replacing constants $-\alpha v/t$, $-\beta v/t$, and $-\gamma v/t$ with $\alpha$, $\beta$, and $\gamma$, we obtain,

[TABLE]

If there exist $\alpha,\beta$, and $\gamma$ satisfying Eq. (47), there must be solutions that Eq. (17) hold. This solution is ZD strategies with errors. This is consistent with Hao et al.’s [49].

Case (2) $\epsilon-\xi=0$ and $1-3\epsilon-\xi=0$:

In this case, the equations $\epsilon-\xi=0$ and $1-3\epsilon-\xi=0$ lead to $\epsilon=1/4$ and $\xi=1/4$. When $\epsilon=1/4$ and $\xi=1/4$, the expected payoffs $R_{E}=1/2(R+S),S_{E}=1/2(R+S),T_{E}=1/2(T+P)$, and $P_{E}=1/2(T+P)$ hold, which do not satisfy the condition of the prisoner’s dilemma game: $T_{E}>R_{E}>P_{E}>S_{E}$. Hence, we can exclude this solution.

Case (3) $p_{1}-p_{2}=0$ and $p_{3}-p_{4}=0$:

In this case, we substitute $p_{1}-p_{2}=0$ and $p_{3}-p_{4}=0$ into Eq. (42) to obtain

[TABLE]

The equations $\mu=0$ and $\eta=0$ do not hold at the same time. The following equations must hold.

[TABLE]

Therefore, we obtain the solution $p_{1}=p_{2}=p_{3}=p_{4}=-u/s$, which is the other solution of Eq. (42). Let $r\ (0\leq r\leq 1)$ be $-u/s$. Next, we check whether this solution satisfies Eq. (41). We substitute $p_{1}=p_{2}=p_{3}=p_{4}=r$ and $u=-sr$ into Eq. (41) to obtain

[TABLE]

There exist real numbers $s,t,u,v,\alpha,\beta$, and $\gamma$ which satisfies Eq. (50) as follows:

[TABLE]

This strategy set is unconditional strategies $\bm{p}=(r,r,r,r),0\leq r\leq 1$. Therefore, the unconditional strategies enforce a linear payoff relationship in the RPD game with errors because there exist real numbers $s,t,u,v,\alpha,\beta$, and $\gamma$ such that Eq. (40) and Eq. (41) are satisfied.

Appendix D The feasible payoff-range for unconditional strategies

In this section, we show the feasible expected payoff-range when a player takes unconditional strategies. We assume that player $X$ takes unconditional strategies, which is $\bm{p}=(r,r,r,r)$. By substituting unconditional strategies $p_{1}=p_{2}=p_{3}=p_{4}=r$ into Eq. (11), we obtain

[TABLE]

The equations $\tau=1-2\epsilon-\xi,\mu=1-\epsilon-\xi$ and $\eta=\epsilon+\xi,\mu+\eta=1$ lead to

[TABLE]

By subtracting $r$ times the third column from the first, we obtain

[TABLE]

By subtracting the third row from the first and the fourth from the second, we obtain

[TABLE]

By subtracting $r$ times the first row from the third and the fourth from $r$ times the second, we obtain

[TABLE]

The Laplace expansion along the first column yields:

[TABLE]

Additionally, the Laplace expansion along the first column yields:

[TABLE]

Therefore $X$’s expected payoff can be calculated by the form of the determinant as follows:

[TABLE]

Let $x$ be $\eta q_{1}+\mu q_{2}-1+r(\mu-\eta)(q_{1}-q_{2})$ and $y$ be $\eta q_{3}+\mu q_{4}+r(\mu-\eta)(q_{3}-q_{4})$ to obtain

[TABLE]

where $-1\leq x\leq 0$ and $0\leq y\leq 1$ because $0\leq q_{1},q_{2},q_{3},q_{4}\leq 1$. In the case of $x\neq 0$, let $k$ be $y/x$, where $-\infty<k\leq 0$ ($\because-1\leq x<0$ and $0\leq y\leq 1$). Then, Eq. (64) leads to

[TABLE]

Here, by the conditions $0\leq r\leq 1$ and $T_{E}>R_{E}>P_{E}>S_{E}$, the $s_{X}$ is maximum if the function $f(k)=1/(1-k)$ is minimum and the $s_{X}$ is minimum if the function $f(k)$ is maximum. Then, the maximum of $f(k)$ is $f(0)=1$ and the minimum do not exist but $\lim_{k\to-\infty}f(k)\approx 0$. Hence, the range of $s_{X}$ in the case of $x\neq 0$ is $r(S_{E}-P_{E})+P_{E}\leq s_{X}<r(R_{E}-T_{E})+T_{E}$. Next, in the case of $x=0$, $s_{X}=r(R_{E}-T_{E})+T_{E}$ holds. From the above, the feasible expected payoff-range for unconditional strategies is given by

[TABLE]

For instance, when $(T,R,P,S)=(1.5,1,0,-0.5)$ is given, the expected payoffs become $R_{E}=1-1.5(\epsilon+\xi),S_{E}=-0.5+1.5(\epsilon+\xi),T_{E}=1.5(1-\epsilon-\xi)$, and $P_{E}=1.5(\epsilon+\xi)$, respectively. If player $X$ is ALLD ($r=0$), his expected payoff becomes $1.5(\epsilon+\xi)\leq s_{X}\leq 1.5(1-\epsilon-\xi)$ by Eq. (66). Thus, when $\epsilon+\xi=0,0.1,0.2$, and 0.3, the ranges become $0\leq s_{X}\leq 1.5$, $0.15\leq s_{X}\leq 1.35$, $0.3\leq s_{X}\leq 1.2$, and $0.45\leq s_{X}\leq 1.05$, respectively.

Moreover, we can even know the possible payoff range for $s_{Y}$ when player $X$ takes $\bm{p}=(r,r,r,r)$ by replacing $\bm{S}_{X}$ with $\bm{S}_{Y}$ in Eq. (59).

Appendix E TFT can enforce a linear payoff relationship under errors only when special conditions are satisfied

In general, with errors, TFT can enforce a linear payoff relationship only when special conditions are satisfied. We prove it in this section.

As we showed in the main text, the only strategies that enforce a linear payoff relationship are either ZD or unconditional strategies with observation errors. Thus, TFT must be included in one of them. It is obvious that TFT is not classified as an unconditional strategy because it is specified by $\bm{p}=(1,0,1,0)$. Therefore, we check whether TFT can be classified as ZD strategies. If there exist strategies that satisfy Eq. (22), TFT is one of the ZD strategies.

By substituting $\bm{p}=(1,0,1,0)$ into Eq. (22), we obtain

[TABLE]

We solve this equation and obtain the following two types of the solution:

[TABLE]

or

[TABLE]

which means that only in the case that there are no errors ($\eta=0\Leftrightarrow\epsilon=0,\xi=0$) as already proven in Appendix B or the case with $R_{E}+P_{E}=T_{E}+S_{E}$, TFT can enforce a linear payoff relationship. Figure E.1 shows the case with $R_{E}+P_{E}\neq T_{E}+S_{E}$. When there are no errors (black dots), TFT can enforce a linear payoff relationship. However, in the other cases (green and light green dots), the linear relationship collapses.

Acknowledgment

This study was partly supported by HAYAO NAKAYAMA Foundation for Science & Technology and Culture and JSPS KAKENHI Grant Number JP19K04903 (G.I.).