Evolutionary dynamics with game transitions

TL;DR

This paper investigates how dynamic changes in the environment, modeled as game transitions, influence the evolution of cooperation in structured populations, revealing that certain transition rules can promote prosocial behavior.

Contribution

It introduces a model of evolutionary dynamics with game transitions and derives a simple rule showing how environment changes can favor cooperation.

Findings

Weak selection favors cooperation if benefit-to-cost ratio exceeds a threshold involving game transition effects.

Game transitions can promote cooperation even when individual games do not favor it.

Small variations in game types can significantly enhance cooperative behavior.

Abstract

The environment has a strong influence on a population's evolutionary dynamics. Driven by both intrinsic and external factors, the environment is subject to continual change in nature. To capture an ever-changing environment, we consider a model of evolutionary dynamics with game transitions, where individuals' behaviors together with the games they play in one time step influence the games to be played next time step. Within this model, we study the evolution of cooperation in structured populations and find a simple rule: weak selection favors cooperation over defection if the ratio of the benefit provided by an altruistic behavior, $b$, to the corresponding cost, $c$, exceeds $k-k'$, where $k$ is the average number of neighbors of an individual and $k'$ captures the effects of the game transitions. Even if cooperation cannot be favored in each individual game, allowing for a transition to a relatively valuable game after mutual cooperation and to a less valuable game after defection can result in a favorable outcome for cooperation. In particular, small variations in different games being played can promote cooperation markedly. Our results suggest that simple game transitions can serve as a mechanism for supporting prosocial behaviors in highly-connected populations.

Equations

$${\rho }_C>{\rho }_D⟺\frac{b_1}{c}>k-\xi \frac{\Delta b}{c},$$

$${\rho }_C>{\rho }_D⟺\xi \frac{\Delta b}{c}>1,$$

$$\xi =\frac{k-1}{2}q-\frac{k+1}{2}p.$$

$${\rho }_C>{\rho }_D⟺\frac{b_1}{c}>k-\sum _{i=2}^n{\xi }_i\frac{\Delta b_{1i}}{c},$$

$${\rho }_C>{\rho }_D⟺\sum _{i=2}^n{\xi }_i\frac{\Delta b_{1i}}{c}>1.$$

$${\xi }_i=\frac{\left(k-1\right)}{2}{u}_i^{\left(1\right)}-\frac{\left(k+1\right)}{2}{u}_i^{\left(2\right)}$$

$${\xi }_i=\frac{u_i^{\left(1\right)}}{2}-\frac{u_i^{\left(2\right)}}{2}$$

$${\xi }_2=\frac{\left(k-1\right)p_{12}^{\left(1\right)}}{2\left(p_{12}^{\left(1\right)}+p_{21}^{\left(1\right)}\right)}-\frac{\left(k+1\right)p_{12}^{\left(2\right)}}{2\left(p_{12}^{\left(2\right)}+p_{21}^{\left(2\right)}\right)}$$

$${\xi }_2=\frac{p_{12}^{\left(1\right)}}{2\left(p_{12}^{\left(1\right)}+p_{21}^{\left(1\right)}\right)}-\frac{p_{12}^{\left(2\right)}}{2\left(p_{12}^{\left(2\right)}+p_{21}^{\left(2\right)}\right)},$$

$$\bordermatrix{&A&B\cr A&R_{i}&S_{i}\cr B&T_{i}&P_{i}\cr},$$

$$\textbf{P}^{(2)}=\left[\begin{array}[]{ccc}p_{11}^{(2)}&\cdots&p_{1n}^{(2)}\\ \vdots&\ddots&\vdots\\ p_{n1}^{(2)}&\cdots&p_{nn}^{(2)}\end{array}\right],\quad\textbf{P}^{(1)}=\left[\begin{array}[]{ccc}p_{11}^{(1)}&\cdots&p_{1n}^{(1)}\\ \vdots&\ddots&\vdots\\ p_{n1}^{(1)}&\cdots&p_{nn}^{(1)}\end{array}\right],\quad\textbf{P}^{(0)}=\left[\begin{array}[]{ccc}p_{11}^{(0)}&\cdots&p_{1n}^{(0)}\\ \vdots&\ddots&\vdots\\ p_{n1}^{(0)}&\cdots&p_{nn}^{(0)}\end{array}\right],$$

$$p_A+p_B=1;$$

$$\sum _{i=1}^n{q}_{X|Y}^{(i)}=q_{X|Y};$$

$$p_{XY}^{(i)}=q_{X|Y}^{(i)}{p}_Y;$$

$$p_{XY}^{(i)}=p_{YX}^{(i)};$$

$$q_{A|X}+q_{B|X}=1;$$

$$p_{XY}=q_{X|Y}{p}_Y.$$

$$\mathcal{B}\left(k_{A|B}^{(i)},k_{B|B}^{(i)}|i=1,\dots ,n\right)=\frac{k!}{{\Pi }_{i=1}^n\left(k_{A|B}^{(i)}!k_{B|B}^{(i)}!\right)}{\Pi }_{i=1}^n\left[{\left(q_{A|B}^{(i)}\right)}^{k_{A|B}^{(i)}}{\left(q_{B|B}^{(i)}\right)}^{k_{B|B}^{(i)}}\right].$$

$$f_{A|B}^{(i)}=1-\delta +\delta \left[(k-1)\sum _{j=1}^n{q}_{A|A}^{(j)}{R}_j+(k-1)\sum _{j=1}^n{q}_{B|A}^{(j)}{S}_j+S_i\right].$$

$$f_{B|B}^{(i)}=1-\delta +\delta \left[(k-1)\sum _{j=1}^n{q}_{A|B}^{(j)}{T}_j+(k-1)\sum _{j=1}^n{q}_{B|B}^{(j)}{P}_j+P_i\right].$$

$$\mathbb{P}\left(A\to B\right)=\frac{{\sum }_{i=1}^n{k}_{A|B}^{(i)}{f}_{A|B}^{(i)}}{{\sum }_{i=1}^n\left(k_{A|B}^{(i)}{f}_{A|B}^{(i)}+k_{B|B}^{(i)}{f}_{B|B}^{(i)}\right)}.$$

$$\mathbb{P}\left(B\to B\right)=\frac{{\sum }_{i=1}^n{k}_{B|B}^{(i)}{f}_{B|B}^{(i)}}{{\sum }_{i=1}^n\left(k_{A|B}^{(i)}{f}_{A|B}^{(i)}+k_{B|B}^{(i)}{f}_{B|B}^{(i)}\right)}.$$

$$\mathbb{P}\left(\Delta p_A=\frac{1}{N}\right)=p_B\sum _{{\sum }_{i=1}^n\left(k_{A|B}^{(i)}+k_{B|B}^{(i)}\right)=k}\mathcal{B}\left(k_{A|B}^{(i)},k_{B|B}^{(i)}|i=1,\dots ,n\right)\mathbb{P}\left(A\to B\right).$$

$$\mathcal{A}\left(k_{A|A}^{(i)},k_{B|A}^{(i)}|i=1,\dots ,n\right)=\frac{k!}{{\Pi }_{i=1}^n\left(k_{A|A}^{(i)}!k_{B|A}^{(i)}!\right)}{\Pi }_{i=1}^n\left[{\left(q_{A|A}^{(i)}\right)}^{k_{A|A}^{(i)}}{\left(q_{B|A}^{(i)}\right)}^{k_{B|A}^{(i)}}\right].$$

$$f_{A|A}^{(i)}=1-\delta +\delta \left[(k-1)\sum _{j=1}^n{q}_{A|A}^{(j)}{R}_j+(k-1)\sum _{j=1}^n{q}_{B|A}^{(j)}{S}_j+R_i\right].$$

$$f_{B|A}^{(i)}=1-\delta +\delta \left[(k-1)\sum _{j=1}^n{q}_{A|B}^{(j)}{T}_j+(k-1)\sum _{j=1}^n{q}_{B|B}^{(j)}{P}_j+T_i\right].$$

$$\mathbb{P}\left(A\to A\right)=\frac{{\sum }_{i=1}^n{k}_{A|A}^{(i)}{f}_{A|A}^{(i)}}{{\sum }_{i=1}^n\left(k_{A|A}^{(i)}{f}_{A|A}^{(i)}+k_{B|A}^{(i)}{f}_{B|A}^{(i)}\right)}.$$

$$\mathbb{P}\left(B\to A\right)=\frac{{\sum }_{i=1}^n{k}_{B|A}^{(i)}{f}_{B|A}^{(i)}}{{\sum }_{i=1}^n\left(k_{A|A}^{(i)}{f}_{A|A}^{(i)}+k_{B|A}^{(i)}{f}_{B|A}^{(i)}\right)}.$$

$$\mathbb{P}\left(\Delta p_A=-\frac{1}{N}\right)=p_A\sum _{{\sum }_{i=1}^n\left(k_{A|A}^{(i)}+k_{B|A}^{(i)}\right)=k}\mathcal{A}\left(k_{A|A}^{(i)},k_{B|A}^{(i)}|i=1,\dots ,n\right)\mathbb{P}\left(B\to A\right).$$

$$\begin{array}{rl}{\dot{p}}_A& =\frac{1}{N}\mathbb{P}\left(\Delta p_A=\frac{1}{N}\right)+\left(-\frac{1}{N}\right)\mathbb{P}\left(\Delta p_A=-\frac{1}{N}\right)\\ & =\delta \frac{1}{kN}\sum _{i=1}^n\left(I_{R_i}{R}_i+I_{S_i}{S}_i+I_{T_i}{T}_i+I_{P_i}{P}_i\right)+O\left({\delta }^2\right),\end{array}$$

$$I_{R_i}=p_{AA}^{(i)}(k-1)\left[(k-1)q_{B|A}(q_{A|A}+q_{B|B})+q_{B|A}\right];$$

$$I_{S_i}=p_{AB}^{(i)}(k-1)\left[(k-1)q_{B|A}(q_{A|A}+q_{B|B})+q_{B|B}\right];$$

$$I_{T_i}=-p_{AB}^{(i)}(k-1)\left[(k-1)q_{A|B}(q_{A|A}+q_{B|B})+q_{A|A}\right];$$

$$I_{P_i}=-p_{BB}^{(i)}(k-1)\left[(k-1)q_{A|B}(q_{A|A}+q_{B|B})+q_{A|B}\right].$$

Full text

1. Introduction

The prosocial act of bearing a cost to provide another individual with a benefit, which is often referred to as “cooperation” [1], reduces the survival advantage of the donor and fosters that of the recipient. Understanding how such a trait can be maintained in a competitive world has long been a focal issue in evolutionary biology and ecology [2]. The spatial distribution of a population makes an individual more likely to interact with neighbors than with those who are more distant. Population structures can affect the evolution of cooperation [3, 4, 5, 6, 7, 8, 9]. In “viscous” populations, one’s offspring often stay close to their places of birth. Relatives thus interact more often than two random individuals. Compared with the well-mixed setting, population “viscosity” is known to promote cooperation [12] (although, when the population density is fixed, local competition can cancel the cooperation-promoting effect of viscosity [10, 11]). Past decades have seen an intensive investigation of the evolution of cooperation in graph-structured populations [6, 7, 8, 9]. One of the best-known findings is that weak selection favors cooperation if the ratio of the benefit provided by an altruistic act, $b$, to the cost of expressing such an altruistic trait, $c$, exceeds the average number of neighbors, $k$, i.e. $b/c>k$ [6, 13]. This simple rule strongly supports the proposition that population structure is one of the major mechanisms responsible for the evolution of cooperation [2].

On the other hand, many realistic systems are highly-connected, with each individual having many neighbors on average. For example, in a contact network consisting of students from a French high school, each student has $36$ neighbors on average, meaning $k=36$ [14]. In such cases, the threshold for establishing cooperation, based on the rule “$b/c>k$,” is quite high: the benefit from an altruistic act must be at least $36$ times greater than its cost. Somewhat large mean degrees have also been observed in collegiate Facebook networks, with well-known examples ranging from $39$ neighbors to well over $100$ [15, 16, 17]. Such networks can (and do) involve the expression of social behaviors much more complex than those captured by the simple model of cooperation described previously. However, even for such a simple model, it is not understood if and when the threshold for the evolution of cooperation can be reduced to something less than the mean number of neighbors. Here, we consider a natural way in which this threshold can be relaxed using “game transitions.”

In evolutionary game theory, an individual’s reproductive success is determined by games played within the population. Many prior studies have relied on an assumption that the environment in which individuals evolve is time-invariant, meaning the individuals play a single, fixed game. However, this assumption is not always realistic and can represent an oversimplification of reality [18], as many experimental studies have shown that the environment individuals face changes over time (and often) [19, 20, 21, 22]. As a simple example, overgrazing typically leads to the degradation of the common pasture land, leaving herders with fewer resources to utilize in subsequent seasons. By constraining the number of livestock within a reasonable range, herders can achieve a more sustainable use of pasture land [23]. In this kind of population, individuals’ actions influence the state of environment, which in turn impacts the actions taken by its members. Apart from endogenous factors like individuals’ actions, exogenous factors like seasonal climate fluctuations and soil conditions can also modify the environment experienced by the individuals. Examples are not limited to human-related activities but also appear in various microbial systems including bacteria and viruses [21, 22].

In this study, we use graphs to model a population’s spatial structure, where nodes represent individuals and edges describe their interactions. We propose a model of evolutionary dynamics with game transitions: individuals sharing an edge interact (“play a game”) in each time step, and their strategic actions together with the game played determine the game to be played in the next time step. We find that game transitions can lower the threshold for establishing cooperation by $k^{\prime}$, which means that the condition for cooperation to evolve is $b/c>k-k^{\prime}$, where $k^{\prime}$ captures the effects of the game transitions. Even if cooperation is disfavored in each individual game, transitions between the games can be favorable for the evolution of cooperation. In fact, just slight differences between games can dramatically lower the barrier for the success of cooperators. Our results suggest that game transitions can play a critical role in the evolution of prosocial behaviors.

2. Model

We study a population of $N$ players consisting of cooperators, $C$, and defectors, $D$. The population structure is described by a graph. Each player occupies a node on the graph. Edges between nodes describe the events related to interactions and biological reproduction (or behavior imitation). In each time step, each player interacts separately with every neighbor, and the games played in different interactions can be distinct (Fig. 1A). When playing game $i$, mutual cooperation brings each player a “reward,” $R_{i}$, whereas mutual defection leads to an outcome of “punishment,” $P_{i}$; unilateral cooperation leads to a “sucker’s payoff,” $S_{i}$, for the cooperator and a “temptation,” $T_{i}$, for the defector. We assume that each game is a prisoner’s dilemma, which is defined by the payoff ranking $T_{i}>R_{i}>P_{i}>S_{i}$. Each player derives an accumulated payoff, $\pi$, from all interactions, and this payoff is translated into reproductive fitness, $f=1-\delta+\delta\pi$, where $\delta\geqslant 0$ represents the intensity of selection [24]. We are particularly concerned with the effects of weak selection [25, 26], meaning $0<\delta\ll 1$.

At the end of each time step, one player is selected for death uniformly at random from the population. The neighbors of this player then compete for the empty site, with each neighbor sending an offspring to this location with probability proportional to fitness. Following this “death-birth” update step, the games played in the population also update based on the previous games played and the actions taken in those games (Fig. 1B). For the player occupying the empty site, the games it will play are determined by the interactions of the prior occupant.

The game transition can be deterministic or stochastic (probabilistic). If the game to be played is independent of the previous game, the game transition is “state-independent” [18]. When the game that will be played depends entirely on the previous game, the game transition is “behavior-independent.” The simplest case is when the games in all interactions are identical initially and remain constant throughout the evolutionary process, which is the setup of most prior studies [6].

3. Results

In the absence of mutation, a finite population will eventually reach a monomorphic state in which all players have the same strategy, either all-cooperation or all-defection. We study the competition between cooperation and defection by comparing the fixation probability of a single cooperator, $\rho_{C}$, to that of a single defector, $\rho_{D}$. Concretely, $\rho_{C}$ is the probability that a cooperator starting in a random location generates a lineage that takes over the entire population. Analogously, $\rho_{D}$ is the probability that a defector in a random position turns a population of cooperators into defectors. Selection favors cooperators relative to defectors if $\rho_{C}>\rho_{D}$ [24].

3.1. Game transitions between two states

We begin with the case of deterministic game transitions between two states. Each state corresponds to a donation game (see SI Appendix, sections 3 and 4 for a comprehensive investigation of two-state games). In game $1$, a cooperator bears a cost of $c$ to bring its opponent a benefit of $b_{1}$, and a defector does nothing. Analogously, in game $2$, a cooperator pays a cost of $c$ to bring its opponent a benefit of $b_{2}$. That is, $R_{i}=b_{i}-c$, $S_{i}=-c$, $T_{i}=b_{i}$, and $P_{i}=0$ in game $i$. Both $b_{1}$ and $b_{2}$ are larger than $c$. The preferred choice for each player is defection, but $R_{i}>P_{i}$ in each game, resulting in the dilemma of cooperation. We say that game $i$ is “more valuable” than game $j$ if $b_{i}>b_{j}$. We take $b_{1}>b_{2}$ and explore a natural transition structure in which only mutual cooperation leads to the most valuable game.

If every player has $k$ neighbors (i.e. the graph is “$k$-regular”), we find that

[TABLE]

where $\Delta b=b_{1}-b_{2}$ and $\xi=\left(k-1\right)/2$. Note that $\xi$ is positive and independent of payoff values such as $b_{1}$, $b_{2}$, and $c$. We obtain this condition under weak selection, based on the assumption that the population size $N$ is much larger than $k$. When $b_{1}=b_{2}$, the two games are the same, which leads to the well-known rule of $b_{1}/c>k$ for cooperation to evolve on regular graphs [6]. The existence of the term $\xi\Delta b/c$ indicates that transitions between different games can reduce the barrier for the success of cooperation. Even when both games oppose cooperation individually, i.e. $b_{1}/c<k$ and $b_{2}/c<k$, transitions between them can promote cooperation (Fig. 2A). Our analytical results agree well with numerical simulations.

The beneficial effects of game transitions on cooperation become more prominent on graphs of large degree, $k$. We find that a slight difference between games $1$ and $2$, $\Delta b$, can remarkably lower the barrier for cooperation to evolve. For example, when $k=100$ and $c=1$, the critical benefit-to-cost ratio, $\left(b_{1}/c\right)^{*}$, decreases from $100$ to $50.5$ for $\Delta b=1.0$ (Fig. 2B). Therefore, game transitions can significantly promote cooperation in realistic and highly-connected societies [27]. We find that similar results hold under the closely-related “imitation” updating (SI Appendix, Fig. S1 and section 3).

Next, we consider “birth-death” [28] and “pairwise-comparison” [29, 30] updating. Under birth-death updating, in each time step a random player is selected for reproduction with probability proportional to fitness. The offspring replaces a random neighbor. Under pairwise-comparison updating, a player is first selected uniformly-at-random to update his or her strategy. When player $i$ is chosen for a strategy updating, it randomly chooses a neighbor $j$ and compares payoffs. If $\pi_{i}$ and $\pi_{j}$ are the payoffs to $i$ and $j$, respectively, player $i$ adopts $j$’s strategy with probability $1/\left[1+\exp\left(-\delta\left(\pi_{j}-\pi_{i}\right)\right)\right]$ and retains its old strategy otherwise. When mutual cooperation leads to game 1 and other action profiles lead to game 2, under both birth-death and pairwise-comparison updating we have the rule

[TABLE]

where $\xi=1/2$ (SI Appendix, sections 3 and 4). When the two games are the same, $\Delta b=0$, and cooperators are never favored over defectors (Fig. 3AC). Game transitions provide an opportunity for cooperation to thrive as long as $b_{1}-b_{2}>c/\xi$, which opens an avenue for the evolution of cooperation under birth-death and pairwise-comparison updating. One can attribute this result to the fact that under this transition structure, mutual cooperation results in $b_{1}-c$ but when two players use different actions, the cooperator gets $-c$ and the defector gets $b_{2}$. If $b_{1}-b_{2}>c/\xi$, then it must be true that $b_{1}-c>b_{2}$, which means that the players are effectively in a coordination game with a preferred outcome of mutual cooperation.

More intriguingly, Eq. 2 shows that the success of cooperators depends on the difference between benefits provided by an altruistic behavior in game $1$ and game $2$, and it is independent of the exact value in each game (Fig. 3BD). Thus, in a dense population where individuals have many neighbors, even if the benefits provided by an altruistic behavior are low in both game 1 and game 2, transitions between them can still support the evolution of cooperation. We stress that the difference between the two games required to favor cooperation is surprisingly small. For example, $b_{1}-b_{2}>2c$ warrants the success of cooperation over defection on graphs of any degree.

We further examine random graphs [31] and scale-free networks [32], where players differ in the number of their neighbors (SI Appendix, Fig. S2). We find that game transitions can provide more advantages for the evolution of cooperation than their static counterparts under death-birth and imitation updating, and they also give a way for cooperation to evolve under birth-death and pairwise-comparison updating. In addition, we study evolutionary processes with mutation and/or behavior exploration (SI Appendix, Fig. S3). The results demonstrate the robustness of the effects of game transitions on the evolution of cooperation.

3.2. Stochastic, state-independent transitions

For more general state-independent transitions between two games, let $p$ and $q$ represent the probabilities of transitioning to game 2 (the less-valuable game) after mutual cooperation and after unilateral cooperation/defection, respectively. Under death-birth updating, the condition for cooperation to be favored over defection follows the format of Eq. 1 with

[TABLE]

The example in Fig. 2 corresponds to $p=0$ and $q=1$. We explore all eight deterministic game transitions in Fig. 4. We see that game transitions promote cooperation only when mutual cooperation leads to a more profitable game 1 and unilateral defection leads to a less profitable game 2 (Fig. 4CD). However, when mutual cooperation leads to a detrimental state 1 and unilateral defection leads to a beneficial state 2, it is more difficult for cooperation to evolve (Fig. 4EF). In particular, whether or not the game transitions affect the evolution of cooperation depends strongly on the transition after mutual cooperation and the transition after unilateral cooperation/defection. For example, in Fig. 4BF, changing the transition following mutual cooperation influences $\left(b_{1}/c\right)^{*}$ considerably. Transitions following mutual defection play a less prominent role (Fig. 4CD).

3.3. Game transitions among $n$ states

We turn now to the general setup of game transitions among $n$ states (i.e. games $1$ through $n$). If two players play game $i$ in the current time step and among them there are $s\in\left\{0,1,2\right\}$ cooperators, they will play game $j$ in the next time step with probability $p_{ij}^{\left(s\right)}$. $s$ is $2$ for mutual cooperation, $1$ for unilateral cooperation/defection, and [math] for mutual defection. In the prior example of the state transitioning to game 1 after mutual cooperation and to game 2 otherwise, we have $n=2$ and $p_{21}^{\left(2\right)}=1$. This setup can recover deterministic or probabilistic transitions, state-dependent or -independent, behavior-dependent or -independent, and the traditional models involving only a single game [6, 13], as specific cases. We assume that all games are donation games (see SI Appendix, section 3 for any two-player, two-strategy game). In game $i$, a cooperator pays a cost of $c$ to bring its opponent a benefit of $b_{i}$. Game $1$ is the most valuable, meaning $b_{1}\geqslant b_{i}$ for every $i$.

Under death-birth updating, we find that

[TABLE]

where, for every $i$, $\Delta b_{1i}=b_{1}-b_{i}$, and $\xi_{i}$ depends on the game transition pattern (i.e. $p_{ij}^{(s)}$) but is independent of the benefit in each game, $b_{i}$, and cost, $c$ (see Appendix A for the calculation of $\xi_{i}$). The term $\sum_{i=2}^{n}\xi_{i}\Delta b_{1i}/c$ captures how game transitions influence this threshold. The effects of game transitions on cooperation actually arise from two sources: the game transition pattern and the variation in different games. $\xi_{i}$ captures the former and $\Delta b_{1i}/c$ the latter.

Importantly, these two components are independent, which makes it easier to understand the role of each. Let $k^{\prime}$ denote $\sum_{i=2}^{n}\xi_{i}\Delta b_{1i}/c$ and let $b$ denote $b_{1}$. We can interpret Eq. 4 intuitively: weak selection favors cooperation if the ratio of the benefit from an altruistic behavior, $b$, to its cost, $c$, exceeds the average effective number of neighbors, $k-k^{\prime}$. Analogously, under birth-death or pairwise-comparison updating, we find that

[TABLE]

We refer the reader to Appendix A for the calculation of $\xi_{i}$.

Our study above assumes that in each time step, games played by any two players are likely to update (“global” transitions). We also consider the case that games in only a fraction of interactions have chance to update. When games to be updated are randomly selected from the whole population, such a game transition can be transformed to the global transition with a modified transition matrix (see SI Appendix, section 3). Therefore, Eqs. 4–5 still predict the evolutionary outcome.

We also study the case in which the games to be updated are spatially correlated, with only those nearby an individual who competes to reproduce being affected (“local” transitions). Under death-birth and pairwise-comparison updating, global and local transitions lead to decidedly different models. We show that, however, the simple rules for cooperation to evolve (Eqs. 4 and 5) still hold provided $\xi_{i}$ is modified (SI Appendix, section 1 and Figs. S4-S6). We give a brief overview of local game transitions in Appendix B.

4. Pure versus stochastic strategies

So far, in every time step each player is either a cooperator or a defector. But the model we propose here has a much broader scope than just two pure strategies. For example, we also investigate the competition between stochastic strategies under game transitions. Let $s_{p}$ denote a stochastic strategy with which, in each time step, a player chooses cooperation with probability $p$ and defects otherwise. $s_{1}$ thus corresponds to a pure cooperator and $s_{0}$ to a pure defector.

We find that the condition for $s_{p}$ being favored by selection over $s_{q}$ still follows the format of Eq. 4 under death-birth updating and Eq. 5 under birth-death or pairwise-comparison updating, provided that $\xi_{i}$ is modified (SI Appendix, section 3). When mutual cooperation leads to a more valuable game and other action profiles lead to a less valuable game, under death-birth updating game transitions lower the threshold for a cooperative strategy (i.e. $s_{p}$ with a large $p$) being favored relative to a less cooperative strategy. We also find that game transitions can favor the evolution of a cooperative stochastic strategy under birth-death and pairwise-comparison updating.

5. Discussion

We consider evolutionary dynamics with game transitions, coupling individuals’ actions with the environment. Individuals’ behaviors modify the environment, which in turn affects the viability of future actions in that environment. We find a simple rule for the success of cooperators in an environment that can switch between an arbitrary number of states, namely $b/c>k-k^{\prime}$, where $k^{\prime}$ exactly captures how game transitions affect the evolution of cooperation. When all environmental states are identical, we recover the rule $b/c>k$ [6].

In a two-action game governed by a single payoff matrix with entries $R$, $S$, $T$, and $P$, the so-called “sigma rule” of Tarnita et al. [33] says that there exists $\sigma$ for which cooperators are favored over defectors if and only if $\sigma R+S>T+\sigma P$. The coefficient $\sigma$, which is independent of the payoffs, captures how the spatial model and its associated update rule affect evolutionary dynamics. For an infinite random regular graph under death-birth updating, $\sigma=\left(k+1\right)/\left(k-1\right)$. When all interactions are governed by a donation game with a donation cost $c$ and benefit $b_{1}$, substituting $R=b_{1}-c$, $S=-c$, $T=b_{1}$ and $P=0$ into the sigma rule gives the condition of cooperation being favored over defection. Intriguingly, Eq. 1 can be phrased in the form of a sigma rule, with $R=b_{1}-c+\left(b_{1}-b_{2}\right)\left(k-1\right)/\left(k+1\right)$, $S=-c$, $T=b_{1}$, and $P=0$. With game transitions, evolution proceeds “as if” all interactions are governed by an effective game with $R=b_{1}-c+\left(b_{1}-b_{2}\right)\left(k-1\right)/\left(k+1\right)$, $S=-c$, $T=b_{1}$, and $P=0$. Compared with the donation game, mutual cooperation brings each player an extra benefit of $\left(b_{1}-b_{2}\right)\left(k-1\right)/\left(k+1\right)$ in this effective game. That is, the game transitions create a situation in which two cooperators play a synergistic game and obtain synergistic benefits (see more discussions in SI Appendix, section 3E).

This intuition also holds for birth-death and pairwise-comparison updating. For a prisoner’s dilemma in a constant environment, weak selection disfavors cooperation in any homogeneous structured population [6, 34]. With game transitions, the synergistic benefit to each cooperator upon their mutual cooperation induces a transformation of the payoff structure. In particular, the synergistic benefit can transform the nature of the interaction from a prisoner’s dilemma to a coordination game with a preferred outcome of mutual cooperation.

The fact that game transitions allow cooperation to evolve is related to the idea of partner-fidelity feedback in evolutionary biology [35, 36]. Partner-fidelity feedback describes that one’s cooperation increases its partner’s fitness, which ultimately feeds back as a fitness increase to the cooperator. Unlike reactive strategies like Tit-for-Tat, this feedback is an automatic process and does not require the partner’s conditional response. In the classic example of grass-endophyte mutualism [37, 38], by producing secondary compounds to protect the grass host, endophytes obtain more nutritional provisioning from the host. By providing nutrients to the endophytes, the grass host is more resistant to herbivores due to the increased delivery of secondary compounds. Similarly, in our study mutual cooperation could generate a synergistic benefit, which in turn promotes the evolution of cooperation.

When mutual cooperation allows for a more profitable game and other actions profiles lead to a less profitable game, a slight difference between games considerably reduces the threshold for the evolution of cooperation. The reason is that although the variation in games might be orders of magnitude smaller than the threshold for establishing cooperation, transitions among such games generate a synergistic benefit upon mutual cooperation that is of the same order of magnitude as the cost of a cooperative act. Since the synergistic benefit partly makes up for the loss from a cooperative act, a slight difference between games makes cooperation less costly. This finding is of significance to understanding large-scale cooperation in many highly-connected social networks. In these networks, an individual can have hundreds of neighbors [27, 39] and cooperators thus face the risk of being exploited by many neighboring defectors. If the environment remains constant, cooperation must be profitable enough to make up for exploitation by defection [6]. Game transitions can act to reduce the threshold for maintaining cooperation considerably.

We also find that game transitions can stabilize cooperation even when mutation or random strategy exploration is allowed. In a constant environment, when a mutant defector arises within a cluster of cooperators, it dilutes the spatial assortment of cooperators and thus hinders the evolution of cooperation [40]. When the environment changes as a result of individuals’ behaviors, although the defecting mutant indeed exploits its neighboring cooperators temporarily, the environment in which this happens deteriorates rapidly. As a result, the temptation to defect is weakened. In a constant environment, selection also favors the establishment of spatial assortment while mutation destroys it continuously. The population state finally reaches a “mutation-selection stationary (MSS)” distribution. But when the environment is subject to transitions, the interaction environment would also be a part of this distribution. In this case, the joint distribution over individuals’ states and games could be described as a “game-mutation-selection stationary (GMSS)” distribution.

Recent years have seen a growing interest in exploring evolutionary dynamics in a changing and/or heterogeneous environments [41, 42, 43, 44, 45, 46, 47, 48, 49, 50]. Our model is somewhat different. First, our study accounts for both exogenous factors and individuals’ behaviors in the change of the environment, modeling general environmental feedback. In addition, the environment that two players face is independent of that of another pair of players. Individuals’ strategic behaviors directly influence the environment in which they evolve, which enables an individual to reciprocate with the opponent in a single interaction through environmental feedback. Therefore, even if cooperators are disfavored in each individual environment, cooperators can still be favored over defectors through environmental reciprocity. Such an effect has never been observed in prior studies where all individuals interact in a homogeneous environment [41, 44]. In those studies, although the environments the individuals face can change with time, at any specific stage the environment is identical for all individuals. When defection is a dominant strategy in each individual environment, defection also dominates cooperation in the context of an ever-changing environment [41, 42, 44]. In a recent work, Hilbe et al. [18] found that individuals can rely on repeated interactions and continuous strategies to achieve environmental reciprocity. Compared with their model, in our setup individuals play a one-shot game with a pure, unconditional strategy. Our model shows that without relying on direct reciprocity and any strategic complexity, game transitions can still promote the evolution of cooperation.

Appendix A. Calculation of $\xi_{i}$

Let $p_{ij}^{\left(s\right)}$ ($i,j\in\{1,2,\cdots,n\}$ and $s\in\left\{0,1,2\right\}$) be the probability that the state transitions from game $i$ to game $j$ after $s$ players cooperate. Let $\mathbf{P}^{\left(s\right)}$ denote a game transition matrix, where $p_{ij}^{\left(s\right)}$ is the element in the $i$th row and $j$th column. We present the formula of $\xi_{i}$ for a class of game transition patterns here and show the calculation of $\xi$ for general transitions in SI Appendix, section 3.

For every $s\in\left\{0,1,2\right\}$, suppose that the Markov chain with state space $\left\{1,2,\dots,n\right\}$ and transition matrix $\mathbf{P}^{\left(s\right)}$ has only one recurrence class (and the states therein are aperiodic). Let $\textbf{u}^{\left(s\right)}=\left(u_{1}^{\left(s\right)},\cdots,u_{n}^{\left(s\right)}\right)$ denote the stationary distribution of this chain, i.e. the solution to $\mathbf{u}^{\left(s\right)}=\mathbf{u}^{\left(s\right)}\mathbf{P}^{\left(s\right)}$ with $\sum_{j=1}^{n}u_{i}^{\left(s\right)}=1$. We have (SI Appendix, section 3)

[TABLE]

for death-birth updating and

[TABLE]

for birth-death or pairwise-comparison updating. In particular, for game transitions between two states, we have

[TABLE]

for death-birth updating and

[TABLE]

for birth-death or pairwise-comparison updating. For other game transitions, the evolutionary dynamics (and thus $\xi_{i}$) may be sensitive to the initial condition, i.e. the initial fractions of various games. We illustrate an example calculation of $\xi_{i}$ in SI Appendix, section 4.

Appendix B. Global versus local game transitions

Our study above assumes that game transitions are an automatic (and exogenous) responses to interactions. Thus, in each time step, the games played by any two players are likely to update (“global” transitions). But when the game transitions are subject to individuals’ willingness to play a game, players could present different tendencies to modify the environments in which they evolve. For example, under death-birth updating, if player $i$ is selected for death, then only $i$’s nearest neighbors compete to reproduce and replace $i$ with an offspring. Compared with those not involved in competition around the vacant site, individuals close to the individual to be replaced have stronger incentives to change the environment they face since this environment affects their success in filling the vacancy. In other words, games induced by the nearest neighbors of the deceased drive the evolution of a system. Therefore, one could impose transitions only on these games, leading to “local” transitions (see Fig. 5A).

Birth-death updating requires competition at the population level, so global and local transitions are identical in this case. For death-birth and pairwise-comparison updating, however, global and local transitions lead to different models. We show that the simple rules for cooperation to evolve (Eqs. 4 and 5) still hold provided $\xi_{i}$ is modified (SI Appendix, sections 1 and 4). Specifically, we consider the following transition pattern: when a game has an opportunity to update, it transitions to a more valuable game 1 after mutual cooperation and to a less valuable game 2 after defection. Under death-birth updating we have $\rho_{C}>\rho_{D}$ if and only if $b_{1}/c>k-\xi\Delta b/c$, where $\xi=\left(6k^{4}-10k^{3}+3k^{2}+6k+2\right)/\left(12k^{3}\right)$ (compared to $\xi=(k-1)/2$ for global transitions). For pairwise-comparison updating, $\rho_{C}>\rho_{D}$ if and only if $\xi\Delta b/c>1$, where $\xi=\left(10k^{2}-4k+1\right)/\left(24k^{2}-12k\right)$ (compared to $\xi=1/2$ for global transitions).

According to the nature of the critical threshold ($b_{1}/c>k-\xi\Delta b/c$ for death-birth updating and $\xi\Delta b/c>1$ for pairwise-comparison updating), global transitions act as a more effective promoter of cooperation than local transitions do (Fig. 5BC). But for both kinds of game transitions, many messages are qualitatively the same: game transitions can promote cooperation (Figs. 2–3, SI Appendix, Fig. S4); game transitions can amplify the beneficial effects of game variations on cooperation (Fig. 2, Fig. 3, SI Appendix, Fig. S5); and game transitions responding to mutual cooperation or unilateral cooperation/defection strongly affect cooperation. We include a more detailed discussion of global versus local game transitions in SI Appendix, section 4.

Acknowledgments

We thank the referees for many helpful comments and for suggesting the connection to partner-fidelity feedback. The authors gratefully acknowledge support from the National Natural Science Foundation of China (grants 61751301, 61533001), the China Scholar Council (grant 201706010277), the Bill & Melinda Gates Foundation (grant OPP1148627), the Army Research Laboratory (grant W911NF-18-2-0265), and the Office of Naval Research (grant N00014-16-1-2914).

SI.1. Evolutionary dynamics with local game transitions

We consider game transitions among $n$ states, described by games $1,2,\dots,n$. The payoff structure of game $i$ is

[TABLE]

where each value corresponds to a payoff derived by a player with a strategy in the column against a player with a strategy in the row. The game transition pattern is described by three matrices, i.e.

[TABLE]

where $p_{ij}^{(s)}$ represents the probability that players play game $j$ in the next time step conditioned on that they play game $i$ in the current time step and there are $s$ $A$-players, where $i,j\in\left\{1,2,\dots,n\right\}$ and $s\in\left\{0,1,2\right\}$.

On graphs or social networks, each player occupies a node. If two players (or nodes occupied by players) are connected by an edge or a social tie, they play a one-shot game in each time step. The main idea of the theoretical analysis is to couple the game played by two connected players and their strategy profiles into edges. Let $E_{XY}^{(i)}$ denote an edge in which the two connected players take strategies $X$ and $Y$ ($X,Y\in\left\{A,B\right\}$), respectively, in game $i\in\left\{1,2,\dots,n\right\}$. For example, in edge $E_{AA}^{(1)}$, both of the players take $A$ strategy and they play game $1$. We then introduce the following variables to describe this evolving system:

$p_{A}$: the frequency of $A$-players;

$p_{B}$: the frequency of $B$-players;

$p_{XY}^{(i)}$: the frequency of edge $E_{XY}^{(i)}$;

$q_{X|Y}^{(i)}$: the probability to find an edge $E_{XY}^{(i)}$ given that one node of this edge is occupied by a $Y-$player;

$p_{XY}$: the frequency of edges that connect an $X$-player and a $Y$-player;

$q_{X|Y}$: the conditional probability to find an $X-$player given that the adjacent node is occupied by a $Y-$player.

Then we have the identities

[TABLE]

Note that players’ strategies and the game they play coevolve throughout the evolutionary process. From the perspective of network dynamics, we need to consider the change in the frequency of nodes occupied by $A$-players and the frequency of edge $E_{XY}^{(i)}$. Based on above identities, we can use $p_{A}$ and $q_{X|Y}^{(i)}$ to describe the whole system. In the following, we study a random regular graph, where each node is linked to other $k$ nodes.

SI.1.1. Interactions

In each time step, each player interacts separately with every neighbor, and the games played in different interactions can be distinct. Each player derives an accumulated payoff, $\pi$, from all interactions, and this payoff is translated into reproductive fitness, $f=1-\delta+\delta\pi$, where $\delta\geqslant 0$ represents the intensity of selection [1]. $\delta$ scales the contribution of games played to one’s fitness/reproductive rates. The assumption of $\delta\ll 1$, termed weak selection, describes that the game plays only a very small role or it represents only one of many factors influencing the overall reproductive rate. Besides, this assumption allows to derive analytical results and has been widely used in evolutionary biology [2, 1]. We focus on the effects of weak selection [3, 4].

SI.1.2. Death-birth updating

Under death-birth updating, in each time step, a random player is selected to die; all neighbors then compete to reproduce and send an offspring to the vacant site (with the probability proportional to fitness) [5]. We can also interpret this updat rule in a social setting: a random player $i$ decides to update his or her strategy; subsequently, he or she adopts a neighbor’s strategy with a probability proportional to the neighbor’s fitness. Local transitions account for the fact that only the nearest neighbors compete for the vacancy. When the environment change is subject to human’s willingness, these neighbors, compared with other players not involved in the competition, are more incentivized to modify the environment (games) in which they evolve. Therefore, under local game transitions, only games played by the nearest neighbors of the dead can update. We first investigate the change in the frequency of $A-$players.

SI.1.2.1. Change in $p_{A}$—updating a B-player

A $B$-player is chosen to die with probability $p_{B}$. Let $k_{A|B}^{(i)}$ denote the number of neighbors who adopt strategy $A$ and play game $i$ with the focal (dead) player. Analogously, $k_{B|B}^{(i)}$ denotes the number of neighbors who adopt strategy $B$ and play game $i$ with the focal player. Therefore, $\sum_{i=1}^{n}\left(k_{A|B}^{(i)}+k_{B|B}^{(i)}\right)=k$. The probability for such a neighborhood configuration is

[TABLE]

The fitness of a neighbor who adopts strategy $A$ and plays game $i$ with the focal player is

[TABLE]

The fitness of a neighbor who adopts strategy $B$ and plays game $i$ with the focal player is

[TABLE]

The probability that one of neighboring $A$-players replaces the vacancy under such a neighborhood configuration is given by

[TABLE]

The probability that one of neighboring $B-$players replaces the vacancy under such a neighborhood configuration is given by

[TABLE]

Therefore, $p_{A}$ increases by $1/N$ with probability

[TABLE]

SI.1.2.2. Change in $p_{A}$—updating a $A$-player

An $A$-player is chosen to die with probability $p_{A}$. Let $k_{A|A}^{(i)}$ denote the number of neighbors who adopt strategy $A$ and play game $i$ with the focal player. Analogously, $k_{B|A}^{(i)}$ denotes the number of neighbors who adopt strategy $B$ and play game $i$ with the focal player. Therefore, $\sum_{i=1}^{n}\left(k_{A|A}^{(i)}+k_{B|A}^{(i)}\right)=k$. The probability for such a neighborhood configuration is

[TABLE]

The fitness of a neighbor who adopts strategy $A$ and plays game $i$ with the focal player is

[TABLE]

The fitness of a neighbor who adopts strategy $B$ and plays game $i$ with the focal player is

[TABLE]

The probability that one of neighboring $A$-players replaces the vacancy under such a neighborhood configuration is given by

[TABLE]

The probability that one of neighboring $B-$players replaces the vacancy under such a neighborhood configuration is given by

[TABLE]

Therefore, $p_{A}$ decreases by $1/N$ with probability

[TABLE]

SI.1.2.3. Change in $p_{A}$

Let us now suppose that one strategy replacement event takes place in one unit of time. The time derivative of $p_{A}$ is given by

[TABLE]

where

[TABLE]

SI.1.2.4. Change in $p_{AA}^{(i)}$

We proceed with the change in the frequency of each type of edge. Note that when a random player $l$ is chosen to die, the edges between (i) $l$ and its nearest neighbors and (ii) $l$’s nearest neighbors and next-nearest neighbors have chance to update (see the description of local game transitions). We stress that the change in $p_{AA}^{(i)}$ is different from that in $p_{A}$. $p_{A}$ does not change when neighboring $A$-players replace the focal $A$-player (the dead $A$-player) or neighboring $B$-players replace the focal $B$-player (the dead $B$-player). However, in the same case $p_{AA}^{(i)}$ likely changes since games in these edges may switch, which changes the edge type.

We first consider the case that a random $B$-player is chosen to die. We take the same neighborhood configuration as we do in Section SI.1.2.1, i.e. $k_{A|B}^{(j)},k_{B|B}^{(j)}$ for $j=1,\dots,n$. The change in $p_{AA}^{(i)}$ results from two parts: the switching of edges connecting the focal $B$-player and its nearest neighbors, and the switching of edges connecting the nearest neighbors and the next-nearest neighbors. Under the given neighborhood configuration, the change in $p_{AA}^{(i)}$ based on the former part is

[TABLE]

Eq. SI.18 describes the edge switching of $E_{BA}^{(j)}\rightarrow E_{AA}^{(i)}$, which occurs when (i) a neighboring $A$-player reproduces and its offspring replaces the dead $B$-player, i.e., $BA\rightarrow AA$; (ii) neighboring $A$-players who plays game $j$ with the dead in the current time step then plays game $i$ in the next time step, $(j)\rightarrow(i)$.

The change in $p_{AA}^{(i)}$ due to edges between the nearest and the next-nearest neighbors is

[TABLE]

Eq. SI.19 indicates that regardless of which neighbor replaces the focal $B$-player, the change in $p_{AA}^{(i)}$ due to the edges between the nearest and next-nearest neighbors remains the same.

Next, we consider the case in which a random $A$-player is chosen to die. We take the same neighborhood configuration as we do in Section SI.1.2.2, i.e., $k_{A|A}^{(j)},k_{B|A}^{(j)}$ for $j=1,\dots,n$. The change in $p_{AA}^{(i)}$ due to edges between the focal $A$-player and its nearest neighbors is

[TABLE]

and

[TABLE]

Eq. SI.20 (resp. Eq. SI.21) captures the case in which a neighboring $A$-player (resp. $B$-player) successfully occupies the vacant site.

The change in $p_{AA}^{(i)}$ due to edges between the nearest and next-nearest neighbors is

[TABLE]

From Eqs. SI.18-SI.22, the time derivative of $p_{AA}^{(i)}$ is given by

[TABLE]

where $\delta_{ij}=1$ if $i=j$ and $\delta_{ij}=0$ otherwise.

SI.1.2.5. Change in $p_{AB}^{(i)}$

When a $B$-player is selected to die and its neighbourhood configuration is the same as that in Section SI.1.2.1, the change in $p_{AB}^{(i)}$ due to edges between the nearest and next-nearest neighbors is

[TABLE]

and

[TABLE]

Eq. SI.24 (resp. Eq. SI.25) captures the case when a neighboring $A$-player (resp. $B$-player) successfully occupies the vacant site.

The change in $p_{AB}^{(i)}$ due to edges between the nearest and next-nearest neighbors is

[TABLE]

When an $A$-player is selected to die and its neighbourhood configuration is the same as that in Section SI.1.2.2, the change in $p_{AB}^{(i)}$ due to edges between the nearest and next nearest neighbors is

[TABLE]

and

[TABLE]

Eq. SI.27 (resp. Eq. SI.28) captures the case when a neighboring $A$-player (resp. $B$-player) successfully occupies the vacant site.

The change in $p_{AB}^{(i)}$ due to edges between the nearest and next-nearest neighbors is

[TABLE]

Analogously, we have

[TABLE]

SI.1.2.6. Change in $p_{BB}^{(i)}$

When a $B$-player is selected to die and its neighbourhood configuration is the same as that in Section SI.1.2.1, the change in $p_{AB}^{(i)}$ due to edges between the nearest and next-nearest neighbors is

[TABLE]

and

[TABLE]

Eq. SI.31 (resp. Eq. SI.32) captures the case when a neighboring $A$-player (resp. $B$-player) successfully occupies the vacant site.

The change in $p_{BB}^{(i)}$ due to edges between the nearest and next-nearest neighbors is

[TABLE]

When an $A$-player is selected to die and its neighbourhood configuration is the same as that in Section SI.1.2.2, the change in $p_{BB}^{(i)}$ due to edges between the nearest and next nearest neighbors is

[TABLE]

The change in $p_{BB}^{(i)}$ due to edges between the nearest and next nearest neighbors is

[TABLE]

The derivative of $p_{BB}^{(i)}$ is

[TABLE]

SI.1.2.7. Different time scales

From Eq. SI.23, we have

[TABLE]

and

[TABLE]

When the intensity of selection is weak ($\delta\ll 1$), $q_{A|A}$ reaches its equilibrium much faster than $p_{A}$ (see Eqs. SI.16,SI.38). Thus, the dynamical system converges quickly onto the slow manifold with $\dot{q}_{A|A}=0$, so we have

[TABLE]

From Eqs. SI.3a-SI.3f and SI.39, we find that for all $X,Y\in\left\{A,B\right\}$, $p_{XY}$ and $q_{X|Y}$ are a function of $p_{A}$.

We define a function $\textbf{A}\left(\textbf{R}^{(s)}\right)$ mapping a set of $(n-1)\times(n-1)$ matrix $\textbf{R}^{(s)}$ to a $3(n-1)\times 3(n-1)$ matrix, given by

[TABLE]

where $\alpha=(k-2)(1-p_{A})$ and $\beta=(k-2)p_{A}$. Then we use $\textbf{P}^{(s)}$ in Eq. SI.2 to define two $(n-1)\times(n-1)$ matrices as follow

[TABLE]

where $s\in\left\{0,1,2\right\}$. Let b denote a column vector with $3(n-1)$ entries: the first $n-1$ entries are $p_{AA}$; the next $n-1$ entries are $p_{AB}$; the last $n-1$ entries are $p_{BB}$. Let v denote a column vector $\left(p_{AA}^{(1)},\dots,p_{AA}^{(n-1)},p_{AB}^{(1)},\dots,p_{AB}^{(n-1)},p_{BB}^{(1)},\dots,p_{BB}^{(n-1)}\right)^{T}$. Combining Eq. SI.39 and $p_{XY}^{(n)}=p_{XY}-\sum_{i=1}^{n-1}p_{XY}^{(i)}$, we can reduce the system of Eqs. SI.23,SI.30,SI.36 to

[TABLE]

For a linear system described by Eq. SI.42, its equilibrium points can be obtained by solving the equation $\left[\textbf{A}\left(\bar{\textbf{P}}^{(s)}\right)-2k^{2}\textbf{I}\right]\textbf{v}+\textbf{A}\left(\tilde{\textbf{P}}^{(s)}\right)\textbf{b}=\textbf{0}$. If for $0<p_{A}<1$, all eigenvalues of $\bar{\textbf{A}}$ are negative numbers or complex numbers with negative real parts, the system is asymptotically stable and has a single equilibrium point given by [6]

[TABLE]

Regardless of the initial state of $\left(p_{AA}^{(1)},\dots,p_{AA}^{(n-1)},p_{AB}^{(1)},\dots,p_{AB}^{(n-1)},p_{BB}^{(1)},\dots,p_{BB}^{(n-1)}\right)^{T}$, the system ultimately approaches to the equilibrium point. In other words, the initial fractions of various games do not affect the evolutionary outcome. We state that none of $\bar{\textbf{A}}$’s eigenvalues can be positive, since this leads to a few terms in v increasing above $1$ or decreasing below [math] [6], which is unrealistic in the current system. But $\bar{\textbf{A}}$ may have zero eigenvalues. In such cases, the system described by Eq. SI.42 has more than one equilibrium point. The initial state of $\left(p_{AA}^{(1)},\dots,p_{AA}^{(n-1)},p_{AB}^{(1)},\dots,p_{AB}^{(n-1)},p_{BB}^{(1)},\dots,p_{BB}^{(n-1)}\right)^{T}$ determines the equilibrium point that the system approaches. That is, the initial fractions of various games influence the evolutionary outcome. In Section 2, we provide an approach to efficiently evaluate the dependence of the evolutionary outcome to the initial fractions of various games.

SI.1.2.8. Diffusion approximation

For given game transition matrices and the initial fractions of various games, by solving Eq. SI.42, we obtain $p_{AA}^{(i)}$, $p_{AB}^{(i)}$, and $p_{BB}^{(i)}$ as functions of $p_{A}$. Substituting $p_{AA}^{(i)}$, $p_{AB}^{(i)}$, and $p_{BB}^{(i)}$ into Eqs. SI.17a-SI.17d and combining with Eq. SI.39, we have

[TABLE]

for $1\leqslant i\leqslant n-1$. We obtain $I_{R_{n}}$, $I_{S_{n}}$, $I_{T_{n}}$ and $I_{P_{n}}$ by separately replacing $v_{i}$ in Eq. SI.44a with $\left(p_{AA}-\sum_{i=1}^{n-1}v_{i}\right)$, $v_{n+i-1}$ in Eqs. SI.44b and SI.44c with $\left(p_{AB}-\sum_{i=1}^{n-1}v_{n+i-1}\right)$, and $v_{2n+i-2}$ in Eq. SI.44d with $\left(p_{BB}-\sum_{i=1}^{n-1}v_{2n+i-2}\right)$.

We consider a one-dimensional diffusion process of the random variable $p_{A}$. Within a short time interval $\Delta t$, we have

[TABLE]

The fixation probability $\phi_{A}(x)$ of $A$-players with initial frequency $p_{A}(t=0)=x$, satisfies the following differential equation [see Eq. (5.2.186) in Ref [7] and detailed derivation]:

[TABLE]

The solution to Eq. SI.46 is [see Eq (5.2.189) in Ref [7]]

[TABLE]

where

[TABLE]

In Eq. SI.48, the third equality holds when $\delta$ is sufficiently small.

SI.1.2.9. Fixation probability

In a population of $B$-players, when a fraction $x$ of $B$-players mutates to $A$-players, the fixation probability of these $A$-players is

[TABLE]

The fixation probability of a fraction $x$ of $B$-players is

[TABLE]

Then the ratio of fixation probabilities is

[TABLE]

For sufficiently small $x$, we have

[TABLE]

Overall, for a sufficiently large population and $x=1/N$, the condition of $A$-players being favored over $B$-players ($\rho_{A}>\rho_{B}$) is

[TABLE]

Eq. SI.53 holds for not only death-birth updating, but also for imitation (see Section 1SI.1.3 for details) and pairwise-comparison (see Section 1SI.1.4 for details) updating. Note that for different updating rules, $I_{R_{i}}$, $I_{S_{i}}$, $I_{T_{i}}$, $I_{P_{i}}$ differ.

SI.1.2.10. The rule $b/c>k-k^{\prime}$

We now turn to donation games. The payoff structure for game $i$ is

[TABLE]

Substituting payoff structures into Eq. SI.53 and using Eqs. SI.17a-SI.17d, we obtain the condition for $\rho_{A}>\rho_{B}$ under death-birth updating, given by

[TABLE]

where

[TABLE]

Furthermore, we have

[TABLE]

Using $\alpha_{1}=-\alpha_{c}/k-\sum_{i=2}^{n}\alpha_{i}$ and denoting $(b_{1}-b_{i})/c$ by $\Delta b_{1i}/c$, we can rewrite Eq. SI.55 as

[TABLE]

where

[TABLE]

Inserting Eqs. SI.44a and SI.44c into Eq. SI.59, we get the formula of $\xi_{i}$ for death-birth updating. Letting $k^{\prime}=\sum_{i=2}^{n}\xi_{i}\Delta b_{1i}/c$ and $b=b_{1}$, we obtain the rule $b/c>k-k^{\prime}$.

SI.1.3. Imitation updating

In each time step, a random player $i$ is selected to evaluate its strategy. This player retains its own strategy or imitates a neighbor’s strategy with probability proportional to fitness. Analyzing the evolutionary process as we do under death-birth updating, we have

[TABLE]

where

[TABLE]

We redefine the function $\textbf{A}\left(\textbf{R}^{(s)}\right)$ to be

[TABLE]

Then the system under imitation updating can be reduced to

[TABLE]

All other variables such as $\alpha,\beta,\bar{\textbf{P}}^{(s)},\tilde{\textbf{P}}^{(s)},\textbf{b},\textbf{v}$ follow those defined for death-birth updating.

For donation games described by Eq. SI.54, we have the condition for $\rho_{A}>\rho_{B}$,

[TABLE]

where

[TABLE]

Solving Eq. SI.63 and inserting $I_{R_{i}},I_{T_{i}}$ in Eq. SI.65, we get the expression for $\xi_{i}$.

SI.1.4. Pairwise-comparison updating

In each generation, a random player $i$ is selected to evaluate its strategy. This player randomly selects a neighbor $j$ and compares payoffs. Player $i$ then adopts $j$’s strategy with probability

[TABLE]

where $\pi_{i}$ and $\pi_{j}$ denote the payoffs of $i$ and $j$, respectively. Otherwise, player $i$ retains its strategy.

Analogously, we have

[TABLE]

where

[TABLE]

We redefine the function $\textbf{A}\left(\textbf{R}^{(s)}\right)$ to be

[TABLE]

Then the system under pairwise-comparison updating can be reduced to

[TABLE]

All other variables such as $\alpha,\beta,\bar{\textbf{P}}^{(s)},\tilde{\textbf{P}}^{(s)},\textbf{b},\textbf{v}$ follow those defined for death-birth updating.

For donation games described by Eq. SI.54, we have the condition for $\rho_{A}>\rho_{B}$,

[TABLE]

where

[TABLE]

By solving Eq. SI.70 and inserting $I_{R_{i}},I_{T_{i}}$ in Eq. SI.72, we get the expression for $\xi_{i}$.

SI.2. Approach to evaluate the sensitivity of evolutionary dynamics to the initial condition

Here, we consider the initial condition, which refers to the initial fractions of various games played in the population. By calculating the eigenvalues of matrix $\left[\textbf{A}\left(\bar{\textbf{P}}^{(s)}\right)-2k^{2}\textbf{I}\right]/(kN)$ in Eq. SI.42 and evaluating the sign of all eigenvalues, we can tell whether or not under a given game transition pattern the evolutionary outcome is sensitive to the initial condition under death-birth updating for local game transitions. Analogously, we can study the matrix $\left[\textbf{A}\left(\bar{\textbf{P}}^{(s)}\right)-2k(k+1)\textbf{I}\right]/(kN)$ in Eq. SI.63 under imitation updating and the matrix $\left[\textbf{A}\left(\bar{\textbf{P}}^{(s)}\right)-(8k-4)\textbf{I}\right]/(kN)$ in Eq. SI.70 under pairwise-comparison updating.

In this section, we provide an alternative approach to determine the dependence of the evolutionary outcome on the initial conditions. Based on the game transition matrix $\textbf{P}^{(2)}$, $\textbf{P}^{(1)}$, and $\textbf{P}^{(0)}$ in Eq. SI.2, we define a Markov chain with a state space $\textbf{E}=\left\{1,2,\dots,3n\right\}$. The probability transition matrix for this Markov chain is given by

[TABLE]

The entry in the $i$th row and the $j$th column of M is the transition probability from state $i$ to state $j$. If the defined random process has only one closed communicating class, the evolutionary outcome is independent of the initial condition, regardless of the update rule. However, if it has more than one such class, the evolutionary outcome is sensitive to the initial condition.

For a random process defined by a state space E and a probability transition matrix M, we can examine its communicating class structure as follows: letting $\bar{\textbf{M}}=\sum_{i=1}^{3n}\textbf{M}^{i}$, the above random process has only one closed communicating class if and only if in $\bar{\textbf{M}}$ there exists at least some $i$ ($1\leqslant i\leqslant 3n$) such that all entries in the $i$th column are positive. The random process has more than one closed communicating classes if and only if in $\bar{\textbf{M}}$ for every $i$ ($1\leqslant i\leqslant 3n$) there exists at least one entry of [math] in the $i$th column.

The sign of each entry in $\bar{\textbf{M}}$, taking $\bar{M}_{ij}$ (the entry in the $i$th row and the $j$th column in $\bar{\textbf{M}}$) for example, actually indicates the transition possibility (not probability) from state $i$ to state $j$ within $3n$-step transitions (less than or equal to $3n$ steps). $\bar{M}_{ij}>0$ means that the system can transition from state $i$ to $j$ in at most $3n$ steps. For $\bar{M}_{ij}=0$, the transition is unlikely to happen within $3n$ steps, which indicates that the system entering into state $i$ can never transition to state $j$. If there exists some $j$ ($1\leqslant j\leqslant 3n$) such that all entries in the $j$th column of $\bar{\textbf{M}}$ are positive, any state can transition to state $j$ and thus state $j$ lies in a closed communicating class. In such a situation, if there is another closed communicating class, any state lying in the second class is unlikely to transition to state $j$, which leads to a contradiction. Therefore, a column of positive entries suggests a single closed communicating class. Similarly, if there is only one closed communicating class, any state can transition to one state of the closed communicating class. Thus, there must exist a column of positive entries. Analogously, we can prove that the absence of a column of positive entries implies the existence of more than one closed communicating class. We provide examples with two states for a better understanding of this approach. The game transition matrices are

[TABLE]

We have

[TABLE]

which gives

[TABLE]

There exist entries of [math] in every column of $\bar{\textbf{M}}$. Thus, there is more than one closed communicating class, and the initial fractions of various games affect the evolutionary outcomes. As a consistency check, we calculate the eigenvalues of $\left[\textbf{A}\left(\bar{\textbf{P}}^{(s)}\right)-2k^{2}\textbf{I}\right]/(kN)$ in Eq. SI.42, which are given by $\lambda_{1}=0$, $\lambda_{2}=-(k-2)/(Nk)$, and $\lambda_{3}=-(2k-2)/(Nk)$. The eigenvalue $\lambda_{1}=0$ confirms the sensitivity of the evolutionary outcome to the initial conditions.

Actually, Eq. SI.74 intuitively shows that the game remains fixed throughout the evolutionary process. According to prior studies about edge-dependent games, the evolution proceeds “as if” all interactions are governed by an “effective” game [8, 9]. This “effective” game corresponds to the averaged of games played in all interactions, which suggests that its payoff structure depends on the fractions of various games. Thus the evolutionary outcome is sensitive to the initial condition, in line with the analysis based on the above approach. In Section 4SI.4.3, we illustrate how to calculate $\xi_{i}$ for such a system.

Then, we present an example with game transition matrices

[TABLE]

Actually, this case corresponds to the game transition pattern used in Fig. 1 in the main text: both taking strategy $A$ leads to game 1 and other strategy profiles lead to game 2. We have

[TABLE]

which gives

[TABLE]

Except for entries in the fifth column, all other entries in $\bar{\textbf{M}}$ are positive. There is only one closed communicating class. The evolutionary outcome therefore is insensitive to the initial condition. As a consistency check, we calculate the eigenvalues of $\left[\textbf{A}\left(\bar{\textbf{P}}^{(s)}\right)-2k^{2}\textbf{I}\right]/(kN)$ in Eq. SI.42, which are given by $\lambda_{1}=-2k/N$, $\lambda_{2}=-2k/N$, and $\lambda_{3}=-2k/N$. The system has a unique equilibrium point and the evolutionary outcome is independent of the initial condition. In Section 4SI.4.3, we illustrate how to calculate $\xi_{i}$ for this system.

We now briefly explain why the closed communicating class of this random process can predict the sensitivity of the evolutionary outcome to the initial condition. The main idea is whether or not the initially-assigned game between two connected players constrains the game they play in the long-term evolutionary process. For example, the transition pattern we illustrate in Eq. SI.74 describes that if two individuals initially play game 1, regardless of their strategic actions, they will play game 1 throughout the process. Obviously, the initial game decides the game they play later. Therefore, the initial condition affects the evolutionary outcome.

We rename the states in E as $\left\{E_{AA}^{(1)},\dots,E_{AA}^{(n)},E_{AB}^{(1)},\dots,E_{AB}^{(n)},E_{BB}^{(1)},\dots,E_{BB}^{(n)}\right\}$, where the $i_{th}$ entry corresponds to the original state $i$. In the following, we show that M actually captures the state transition of an edge throughout the process. As defined in Section 1, the state of an edge is given by $E_{XY}^{(i)}$, where $X,Y\in\left\{A,B\right\}$ and $i\in\left\{1,2,\dots,n\right\}$. $X$ and $Y$ are strategies of the two connected players and $i$ is the game they play. The transition of an edge state can arise from two parts: the change in players’ strategies and the change in the game they play. The illustrated below is the transition of an edge in one time step:

[TABLE]

Let $AA$ denote both players taking $A$-strategies, $AB$ denote one player taking $A-$strategy and the other taking $B$-strategy, and $BB$ denote both players taking $B-$strategies. Note that in the current model, in each generation, only a player has the opportunity to update its strategy. Thus, the strategy transition follows (i) $AA$ can remain in $AA$ or transition to $AB$ but can not transition to $BB$; (ii) $AB$ can remain in $AB$ or transition to $AA$ or $BB$; (iii) $BB$ can remain in $BB$ or transition to $AB$ but can not transition to $AA$. That is, for any $i$ and $l$, $E_{AA}^{(i)}$ and $E_{BB}^{(l)}$ are unlikely to transition to each other, corresponding to the two null matrices in M (see 0 in Eq. SI.73). The transition of the game is governed by $\textbf{P}^{(2)}$, $\textbf{P}^{(1)}$, and $\textbf{P}^{(0)}$. For example, $\textbf{P}^{(2)}$ determines whether or not an edge of $E_{AA}^{(i)}$ can transition to $E_{AA}^{(i_{1})}$ or $E_{AB}^{(j)}$ in a time step ($i_{1},j\in\{1,2,\dots,n\}$), corresponding to those terms including $\textbf{P}^{(2)}$ in M. Note that the realistic evolutionary process is much more complicated and it is impossible to obtain the exact transition probability of an edge from one state to another, but the matrix M can describe the possibility that an edge of $E_{X_{1}Y_{1}}^{(i)}$ transitions to that of $E_{X_{2}Y_{2}}^{(j)}$ in a time step. The zero entries in M indicate the transition can never happen and the nonzero entries suggest the transition is can happen.

When the random process has only closed communicating class, let c denote the set of all states lying in this class. Eqs. SI.16,SI.23,SI.30,SI.36 show that for a sufficiently small $\delta$, the fractions of various edges change much faster than the fractions of the two strategies. That is, although there is a frequent transition between $A$-players and $B$-players, namely an $A$-player transitioning to a $B$-player or a $B$-player transitioning to an $A$-player, the fraction of $A$-players varies at a relatively low rate. In the evolutionary process, an edge transitions among various states as the strategies adopted by the connected individuals and the game they play change. The state transition possibility of an edge is described by M. Eventually, the state of this edge enters into the closed communicating class c and can never escape from c, regardless of its initial state. Thus, if the random process defined above has only one closed communicating class, the evolutionary outcome is independent of the initial condition.

When the random process has $m$ ($>1$) closed communicating classes, we denote them by $\textbf{c}_{1},\textbf{c}_{2},\dots,\textbf{c}_{m}$. If it is possible for an edge of $E_{X_{1}Y_{1}}^{(i)}$ to transition to $E_{X_{2}Y_{2}}^{(j)}$ within one update step, we denote it by $E_{X_{1}Y_{1}}^{(i)}\rightarrow E_{X_{2}Y_{2}}^{(j)}$. We apply two propositions below:

(i) for $E_{X_{1}Y_{1}}^{(i_{1})}\in\textbf{c}_{j_{1}}$ and $E_{X_{2}Y_{2}}^{(i_{2})}\in\textbf{c}_{j_{2}}$, $i_{1}\neq i_{2}$ for any $j_{1}\neq j_{2}$ regardless of $X_{1}Y_{1}$ and $X_{2}Y_{2}$;

(ii) in every closed communicating class $\textbf{c}_{j}$, there exists some $i$ satisfying $E_{AA}^{(i)}\in\textbf{c}_{j}$, $E_{AB}^{(i)}\in\textbf{c}_{j}$ and $E_{BB}^{(i)}\in\textbf{c}_{j}$. For different closed communicating classes, $i$ can be different.

About proposition (i), for $E_{AA}^{(i)}\in\textbf{c}_{j}$, since $E_{AA}^{(i)}$ is a recurrent state, there exists $l$ making either $E_{AA}^{(l)}\rightarrow E_{AA}^{(i)}$ or $E_{AB}^{(l)}\rightarrow E_{AA}^{(i)}$. Note that $l$ can be $i$. $E_{BB}^{(l)}\rightarrow E_{AA}^{(i)}$ is impossible since in each generation just one player has the change to update its strategy. $E_{AA}^{(l)}\rightarrow E_{AA}^{(i)}$ means $E_{AA}^{(l)}\rightarrow E_{AB}^{(i)}$ due to the strategy transition, leading to $E_{AB}^{(i)}\in\textbf{c}_{j}$. $E_{AB}^{(l)}\rightarrow E_{AA}^{(i)}$ means $E_{AB}^{(l)}\rightarrow E_{AB}^{(i)}$ due to the game transition, leading to $E_{AB}^{(i)}\in\textbf{c}_{j}$. Thus, $E_{AA}^{(i)}\in\textbf{c}_{j}$ always gives $E_{AB}^{(i)}\in\textbf{c}_{j}$. Similarly, $E_{BB}^{(i)}\in\textbf{c}_{j}$ gives $E_{AB}^{(i)}\in\textbf{c}_{j}$. That is, $E_{X_{1}Y_{1}}^{(i_{1})}\in\textbf{c}_{j_{1}}$ means $E_{AB}^{(i_{1})}\in\textbf{c}_{j_{1}}$ and $E_{X_{2}Y_{2}}^{(i_{2})}\in\textbf{c}_{j_{2}}$ means $E_{AB}^{(i_{2})}\in\textbf{c}_{j_{2}}$. For $j_{1}\neq j_{2}$, $i_{1}\neq i_{2}$. Otherwise, an edge state lies in two different closed communicating classes, which leads to a contradiction.

Based on the proof of proposition (i), every closed communicating class includes at lease one state with the form of $E_{AB}^{(i)}$. Note that $i$ in different closed communicating classes is different. Let $E_{AB}^{(i)}\in\textbf{c}_{j}$. Due to the game transition, there must exist $l$ with $E_{AB}^{(i)}\rightarrow E_{AB}^{(l)}$ and $E_{AB}^{(l)}\in\textbf{c}_{j}$. In addition, due to the strategy transition, we have $E_{AB}^{(i)}\rightarrow E_{AA}^{(l)}$ and $E_{AB}^{(i)}\rightarrow E_{BB}^{(l)}$, which gives $E_{AA}^{(l)}\in\textbf{c}_{j}$ and $E_{BB}^{(l)}\in\textbf{c}_{j}$. Overall, $\textbf{c}_{j}$ includes $E_{AA}^{(l)}$, $E_{AB}^{(l)}$ and $E_{BB}^{(l)}$.

Proposition (i) stresses that when an edge transitions into a state that lies in a closed communicating class like $\textbf{c}_{j_{1}}$, the games to be played by the two connected players are limited by $\textbf{c}_{j_{1}}$. If the edge transitions into a state in another closed communicating class like $\textbf{c}_{j_{2}}$, the two connected players can only play games limited by $\textbf{c}_{j_{2}}$. In particular, the games limited by $\textbf{c}_{j_{1}}$ and those by $\textbf{c}_{j_{2}}$ are completely different. The initial condition affects which closed communicating class an edge will transition into. A representative example is that for $E_{AA}^{(i_{1})}\in\textbf{c}_{j_{1}}$, $E_{AB}^{(i_{1})}\in\textbf{c}_{j_{1}}$, $E_{BB}^{(i_{1})}\in\textbf{c}_{j_{1}}$ and $E_{AA}^{(i_{2})}\in\textbf{c}_{j_{2}}$, $E_{AB}^{(i_{2})}\in\textbf{c}_{j_{2}}$, $E_{BB}^{(i_{2})}\in\textbf{c}_{j_{2}}$, when all players play game $i_{1}$ initially, the games to be played are limited by $\textbf{c}_{j_{1}}$ throughout the process. However, if initially all players play game $i_{2}$, $\textbf{c}_{j_{2}}$ constrains the games to be played throughout the process.

We examine the above approach with $10^{8}$ numerical examples. In every example, we generate three $4\times 4$ random matrices in which all entries are nonnegative. We normalize these matrices to make the sum of entries in each row $1$. The three matrices are assigned to $\textbf{P}^{(2)}$, $\textbf{P}^{(1)}$, and $\textbf{P}^{(0)}$. On the one hand, based on $\textbf{P}^{(2)}$, $\textbf{P}^{(1)}$, and $\textbf{P}^{(0)}$, we calculate the eigenvalues of $\left[\textbf{A}\left(\bar{\textbf{P}}^{(s)}\right)-2k^{2}\textbf{I}\right]/(kN)$ in Eq. SI.42 and record whether or not there are zero eigenvalues. On the other, based on $\textbf{P}^{(2)}$, $\textbf{P}^{(1)}$, and $\textbf{P}^{(0)}$, we calculate $\bar{\textbf{M}}$ and record whether or not there is some $i$ such that all entries in the $i$th column of $\bar{\textbf{M}}$ are positive. In all examples, whenever there is zero entry in each column of $\bar{\textbf{M}}$, there are zero eigenvalues. As long as there exists some $i$ such that all entries in the $i$th column of $\bar{\textbf{M}}$ are positive, there is no zero eigenvalue. Thus, our approach predicts well the sensitivity of the evolutionary outcomes to the initial condition. Furthermore, for $\textbf{P}^{(2)}$, $\textbf{P}^{(1)}$, and $\textbf{P}^{(0)}$ under which the evolutionary outcome is sensitive to the initial condition, a slight perturbation or noise to the game transition pattern (to $\textbf{P}^{(2)}$, $\textbf{P}^{(1)}$, and $\textbf{P}^{(0)}$) can turn this system into one insensitive to the initial condition. A simple way to achieve this is adding to each entry in $\textbf{P}^{(2)}$, $\textbf{P}^{(1)}$, and $\textbf{P}^{(0)}$ an arbitrary small number $\delta_{1}$, $\delta_{2}$, and $\delta_{3}$, respectively, where $\delta_{1}$, $\delta_{2}$, and $\delta_{3}$ are not necessary identical.

SI.3. Evolutionary dynamics with global game transitions

SI.3.1. Global game transitions

In this section, we study evolutionary dynamics with global game transitions. In each time step, games in all interactions have chances to update. We proceed with the mathematical analysis as we do in Section SI.1. We take the same variables and notations (Eqs. SI.2-SI.3f). The change in $p_{A}$ follows Eqs. SI.16-SI.17d. We then investigate the change in the frequency of each type of edge. Assuming that a random $B$-player is selected to die, the change in $p_{AA}^{(i)}$ arises from two parts: the switching of edges connecting the focal (dead) $B$-player and its nearest neighbors, the switching of all other edges. Under the neighborhood configuration given in Section SI.1.2.1, Eq. SI.18 shows the change in $p_{AA}^{(i)}$ due to the former part. The change in $p_{AA}^{(i)}$ due to the latter is

[TABLE]

Suppose that a random $A$-player is selected to die. Under the neighborhood configuration given in Section SI.1.2.2, Eqs. SI.20 and SI.21 capture the change in $p_{AA}^{(i)}$ due to edges between the focal $A$-player and its nearest neighbors. The change in $p_{AA}^{(i)}$ due to the switching of other edges is

[TABLE]

From Eqs. SI.18,SI.20,SI.21,SI.81,SI.82, we obtain the time derivative of $p_{AA}^{(i)}$, given by

[TABLE]

Analogously, we obtain the time derivatives of $p_{AB}^{(i)}$ and $p_{BB}^{(i)}$, given by

[TABLE]

and

[TABLE]

Analogous to Eqs. SI.37 and SI.38, a further analysis to Eq. SI.83 gives Eq. SI.39. We redefine the function $\textbf{A}\left(\textbf{R}^{(s)}\right)$ to be

[TABLE]

where $\mu=kN$. $\alpha$ and $\beta$ follow those defined in Eq. SI.40. We can reduce the system of Eqs. SI.83-SI.85 to

[TABLE]

where $\bar{\textbf{P}}^{(s)},\tilde{\textbf{P}}^{(s)},\textbf{b},\textbf{v}$ are defined in Eqs. SI.41 and SI.42. Similarly, solving Eq. SI.87, substituting the solutions into Eqs. SI.44a-SI.44d, and inserting $I_{R_{i}},I_{S_{i}},I_{T_{i}},I_{P_{i}}$ into Eq. SI.53, we obtain the condition of one strategy being favored over the other under general two-strategy games. In particular, when all games are donation games, Eq. SI.58 predicts when $A$ is favored over $B$. $\xi_{i}$ can be obtained by inserting $I_{R_{i}}$ and $I_{T_{i}}$ into Eq. SI.59. The approach proposed in Section SI.2 still determines whether or not the evolutionary dynamics is sensitive to the initial condition under global game transitions.

SI.3.2. A class of game transition patterns

We provide an alternative approach to study a class of game transition patterns (denoted by $\Omega$): for every $s\in\{0,1,2\}$, in a Markov chain $\{M^{(s)}_{t},t=0,1,2,3,\dots\}$ with state space $\{1,2,\dots,n\}$ and state transition matrix $\mathbf{P}^{(s)}$, there is only one recurrent equivalence class (and the states therein are aperiodic). Introducing $\textbf{u}^{(s)}=\left(u_{1}^{(s)},\dots,u_{n}^{(s)}\right)$, the solution to $\mathbf{u}^{(s)}=\mathbf{u}^{(s)}\mathbf{P}^{(s)}$ with $\sum_{j=1}^{n}u_{i}^{(s)}=1$ is the limiting distribution of this Markov chain.

In each time step, only one among $N$ players has the chance to modify its strategy whereas all games are likely to update. The evolutionary rate of the game in an interaction (or in an edge) is $N/2$ times as large as that of interactants’ strategies. Therefore, for sufficiently large population size $N$, the fractions of various games reach a stationary distribution much faster than the fractions of various strategies. For games between two $A$-players, the stationary distribution is $\mathbf{u}^{(2)}$. Thus, in the interaction of two $A$-players, the expected payoff for each $A$-player is $\sum_{i=1}^{n}u_{i}^{(2)}R_{i}$. Analogously, the stationary distribution for games between an $A$-player and a $B$-player is $\mathbf{u}^{(1)}$. The expected payoff for $A$-player is $\sum_{i=1}^{n}u_{i}^{(1)}S_{i}$ and that of $B$-player is $\sum_{i=1}^{n}u_{i}^{(1)}T_{i}$. For games between two $B$-players, the stationary distribution is $\textbf{u}^{(0)}$. The expected payoff for each defector is $\sum_{i=1}^{n}u_{i}^{(0)}P_{i}$. The game transition creates a situation “as if” all players play an “effective” game, with payoff structure

[TABLE]

which holds for death-birth, imitation, pairwise-comparison, and birth-death updating.

Under death-birth updating, in Eq. SI.53, replacing $R_{i}$, $S_{i}$, $T_{i}$ and $P_{i}$ with $\bar{R}$, $\bar{S}$, $\bar{T}$ and $\bar{P}$, then inserting Eqs. SI.44a-SI.44d, we reduce Eq. SI.53 to

[TABLE]

For donation games with $R_{i}=b_{i}-c$, $S_{i}=-c$, $T_{i}=b_{i}$ and $P_{i}=0$, Eq. SI.89 is further reduced to Eq. SI.58 with

[TABLE]

Similarly, under imitation updating, we can reduce Eq. SI.53 to

[TABLE]

A further analysis of donation games leads to Eq. SI.64 with

[TABLE]

Under both pairwise-comparison and birth-death updating, $A$-players are favored over $B$-players if and only if

[TABLE]

A simplification for donation games gives Eq. SI.71 with

[TABLE]

In this section, we solve $\mathbf{u}^{(s)}=\mathbf{u}^{(s)}\mathbf{P}^{(s)}$ and use the limiting distribution $\mathbf{u}^{(s)}$ to approximate the evolutionary process. Eqs. SI.83-SI.85 actually imply this idea. For weak selection ($\delta\ll 1$), $p_{AA}^{(i)}$, $p_{AB}^{(i)}$, $p_{BB}^{(i)}$ reach the equilibrium point much faster than $p_{A}$ (see Eqs. SI.16 and SI.83-SI.85). The dynamical system thus converges quickly onto the slow manifold with $\dot{p}_{AA}^{(i)}=0$, $\dot{p}_{AB}^{(i)}=0$, $\dot{p}_{BB}^{(i)}=0$. In the righthand of Eq. SI.83, $1/N$ occurs in the third and fourth terms. For a sufficiently large population size $N$ ($N\gg 1$), the existence of $1/N$ may make the two terms negligible relative to the first and second terms. This inspires the idea of using $\sum_{j=1}^{n}p_{AA}^{(j)}p_{ji}^{(2)}-p_{AA}^{(i)}=0$ to approximate $\dot{p}_{AA}^{(i)}=0$. Replacing $p_{AA}^{(j)}$ with $u_{j}^{(2)}$, we have $\mathbf{u}^{(2)}\mathbf{P}^{(2)}=\mathbf{u}^{(2)}$. The analogous analysis to Eqs. SI.83 and SI.84 gives $\mathbf{u}^{(1)}\mathbf{P}^{(1)}=\mathbf{u}^{(1)}$ and $\mathbf{u}^{(0)}\mathbf{P}^{(0)}=\mathbf{u}^{(0)}$.

SI.3.3. Game transitions in a fraction of interactions

With global game transitions, games in all interactions have chances to update in each time step. With local game transitions, games in a fraction of interactions have chances to update in each time step. Note that with local game transitions, the interactions allowing for game transitions are spatially correlated—only games in interactions involved with the deceased individual’s neighbors are likely to update. In this section, we assume that in each time step a fraction $p$ ($0<p<1$) of games are randomly selected to update. In other words, in each interaction, the game has chance to update with probability $p$ and has no chance to update with probability $1-p$. Equivalently, in each interaction, with probability $p$ the game transitions based on the probability matrix $\mathbf{P}^{(s)}$, and with probability $1-p$ the game transitions to itself. Such a situation corresponds to game transitions based on a new probability matrix $\hat{\mathbf{P}}^{(s)}$,

[TABLE]

Note that the solution to $\mathbf{u}^{(s)}=\mathbf{u}^{(s)}\hat{\mathbf{P}}^{(s)}$ is the same as $\mathbf{u}^{(s)}=\mathbf{u}^{(s)}\mathbf{P}^{(s)}$. The setting of a fraction of games being transitioned thus leads to the same results as global transition does.

SI.3.4. Stochastic strategies

Up to this point, all players are assumed to be using pure strategies, namely a player cooperating (taking $A$) unconditionally or a player defecting (adopting $B$) unconditionally in each time step. In the following, we further investigate the case where players take stochastic strategies, i.e. choosing to cooperate with a probability and to defect otherwise. Let $s_{p}$ and $s_{q}$ denote two stochastic strategies. Players taking $s_{p}$ choose to cooperate with probability $p$ and defect with probability $1-p$. If taking $s_{q}$, players choose cooperation with probability $q$ and defection with probability $1-q$. For two connected players, before one of them has a chance to update its strategy (i.e. $s_{p}$ or $s_{q}$), their actions (i.e. cooperation or defection) and games they played update many times. Therefore, in a sufficiently large population, the fractions of various interaction scenarios (consisting of two actions and the game they play) reach a stationary distribution much faster than the fractions of various strategies.

We study the competition between $s_{p}$ and $s_{q}$. By taking $p=1$ (pure cooperators) and $q=0$ (pure defectors), this model can recover the case of pure strategies. In the following, we calculate the stationary distribution of interaction scenarios between players taking $s_{p}$ and players taking $s_{q}$. Let $u_{i,r1,r2}^{(pq)}(t)$ denote the probability that in time $t$ a player with $s_{p}$ chooses action $r_{1}$, a player with $s_{q}$ chooses action $r_{2}$, and they play game $i$, where $i\in\{1,\dots,n\}$ and $r_{1},r_{2}\in\{0,1\}$ ([math] represents defection and $1$ means cooperation). Then, we have

[TABLE]

For the game transition pattern $\Omega$, there exists a stationary distribution and we denote it by $\textbf{u}^{(pq)}=(u_{1,1,1}^{(pq)},\dots,u_{n,1,1}^{(pq)},u_{1,1,0}^{(pq)}$, $\dots,u_{n,1,0}^{(pq)},u_{1,0,1}^{(pq)},\dots,u_{n,0,1}^{(pq)},u_{1,0,0}^{(pq)},\dots,u_{n,0,0}^{(pq)})$, where $u_{i,r_{1},r_{2}}^{(pq)}$ indicates the stationary fraction of interactions in which two players play game $i$ and the one with strategy $s_{p}$ chooses action $r_{1}$ and the other with strategy $s_{q}$ chooses action $r_{2}$. We rewrite Eqs. SI.95a-SI.95d as

[TABLE]

We can get the stationary distribution $\textbf{u}^{(pq)}$ by the left eigenvector with $\sum_{r_{1}=0}^{1}\sum_{r_{2}=0}^{1}\sum_{i=1}^{n}u_{i,r_{1},r_{2}}^{pq}=1$. Actually, letting $\bar{\textbf{u}}^{(pq)}=(u_{1}^{(pq)},\dots,u_{n}^{(pq)})$ and solving

[TABLE]

with $\sum_{i=1}^{n}u_{i}^{pq}=1$, we have $u_{i,1,1}^{(pq)}=pqu_{i}^{(pq)}$, $u_{i,1,0}^{(pq)}=p(1-q)u_{i}^{(pq)}$, $u_{i,0,1}^{(pq)}=(1-p)qu_{i}^{(pq)}$, and $u_{i,0,0}^{(pq)}=(1-p)(1-q)u_{i}^{(pq)}$. In the interaction between a player with strategy $s_{p}$ and a player with strategy $s_{q}$, the former’s expected payoff is

[TABLE]

Under death-birth updating, the condition for strategy $s_{p}$ to be favored over $s_{q}$ (i.e. $\rho_{s_{p}}>\rho_{s_{q}}$) is

[TABLE]

We say that a stochastic strategy is more cooperative if players with such a strategy choose cooperation with a larger probability. That is, for $p>q$, $s_{p}$ is more cooperative than $s_{q}$. For donation games described by Eq. SI.54, Eq. SI.99 can be reduced to Eq. SI.58 with

[TABLE]

Similarly, under imitation updating in donation games, natural selection favors $s_{p}$ over $s_{q}$ if Eq. SI.64 holds, where

[TABLE]

For birth-death or pairwise-comparison updating in donation games, $s_{p}$ is favored over $s_{q}$ if Eq. SI.71 holds, where

[TABLE]

SI.3.5. Intuition based on “sigma rule”

Here we provide a few new insights into how game transitions affect the evolution of $A$-players. In the game between $A$-players and $B$-players governed by a single payoff matrix

[TABLE]

Tarnita et al have found that selection favors $A$-players over $B$-players if and only if

[TABLE]

which is termed as “sigma rule” [10]. The coefficient $\sigma$ captures how the spatial model and its associated update rule affect evolutionary dynamics, whereas is independent of the payoffs. For an infinite random regular graph under death-birth updating, $\sigma=\left(k+1\right)/\left(k-1\right)$.

When all interactions are governed by a fixed donation game with a donation cost $c$ and benefit $b_{1}$, substituting $R=b_{1}-c$, $S=-c$, $T=b_{1}$ and $P=0$ into the sigma rule gives the condition of $A$-players being favored over $B$-players. Intriguingly, Eq. SI.58 can be phrased in the form of a sigma rule with $R=b_{1}-c+\frac{2}{k+1}\sum_{i=2}^{n}\xi_{i}\Delta b_{1i}$, $S=-c$, $T=b_{1}$ and $P=0$. With game transitions, evolution proceeds “as if” all interactions are governed by an effective game with payoff matrix

[TABLE]

Compared with the fixed donation game, mutual cooperation brings each player an extra payoff $\frac{2}{k+1}\sum_{i=2}^{n}\xi_{i}\Delta b_{1i}$ in this effective game. This payoff depends on two factors: game transition patterns (described by $\xi_{i}$), variations in different games (described by $\Delta b_{1i}$).

For an infinite random regular graph under pairwise-comparison updating, $\sigma=1$. Analogously, Eq. SI.71 can be phrased in the form of a sigma rule with $R=b_{1}-c+2\sum_{i=2}^{n}\xi_{i}\Delta b_{1i}$, $S=-c$, $T=b_{1}$ and $P=0$. With game transitions, evolution proceeds “as if” all interactions are governed by an effective game with payoff structure

[TABLE]

SI.4. Representative examples

In Section SI.1 and SI.3, we derive the general condition of one strategy to be favored over the other strategy, which requires to solve a set of equations. In this section, we study four representative interaction scenarios and provide explicit expressions.

SI.4.1. Evolutionary dynamics with state-independent game transitions

If the game to be played in the next time step is independent of the game played in the past, the game transition is state-independent. That is, $p_{im}^{(s)}=p_{jm}^{(s)}$ for all $i$ and $j$. The number of $A$-players determines the game to be played. Let $p_{im}^{(s)}=p_{m}^{(s)}$. For local game transitions, focusing on game transition patterns under which the evolutionary outcome is independent of the initial condition, we have

[TABLE]

For global game transitions, focusing on the game transition pattern $\Omega$ introduced in Section SI.3, we have

[TABLE]

In particular, if $p_{m}^{(s)}=1/n$ for all $m$ and $s$, the game transition is fully stochastic. In the next time step, any game occurs with the equal probability. For both local and global game transitions, weak selection favors $A$ over $B$ if

[TABLE]

Let $\bar{R}=\left(\sum_{i=1}^{n}R_{i}\right)/n$, $\bar{S}=\left(\sum_{i=1}^{n}S_{i}\right)/n$, $\bar{T}=\left(\sum_{i=1}^{n}T_{i}\right)/n$, $\bar{P}=\left(\sum_{i=1}^{n}P_{i}\right)/n$. We find that the evolutionary process with stochastic and diverse games can be approximated by that of a static and unified game.

SI.4.2. Evolutionary dynamics with strategy-independent game transitions

If the game to be played in the next time step is independent of players’ strategic actions in the past, the game transition is strategy-independent. That is, $\textbf{P}^{(2)}=\textbf{P}^{(1)}=\textbf{P}^{(0)}$. Let $\textbf{P}^{(2)}=\textbf{P}^{(1)}=\textbf{P}^{(0)}=\textbf{P}$. Here, we consider the game transition pattern $\Omega$ introduced in Section SI.3.

For global game transitions, we introduce a vector $\textbf{u}=\left(u_{1},u_{2},\dots,u_{n}\right)$, which satisfies $\textbf{u}=\textbf{uP}$. Then, we have

[TABLE]

Note that in pairwise-comparison or birth-death updating, $\xi_{i}=0$ means that cooperation can never evolve regardless of the benefit provided by a cooperative behavior in the donation game. In other words, if the game is independent of strategic actions, game transitions cannot promote cooperation.

SI.4.3. Evolutionary dynamics with game transitions between two states ($n=2$)

Given the theoretical significance of two states, we provide a systematic investigation of game transitions between two donation games. According to Eq. SI.58, under death-birth updating, the general rule for cooperation to be favored over defection is

[TABLE]

For local game transitions, focusing on game transition patterns under which the evolutionary outcome is insensitive to the initial condition, we have

[TABLE]

where

[TABLE]

In the above equations, $P_{2}=p_{11}^{(2)}-p_{21}^{(2)}$, $P_{1}=p_{11}^{(1)}-p_{21}^{(1)}$, and $P_{0}=p_{11}^{(0)}-p_{21}^{(0)}$.

For global game transitions, focusing on the game transition pattern $\Omega$ introduced in Section SI.3, we have

[TABLE]

We proceed with a specific game transition pattern involving two states: mutual cooperation leads to game 1 and all other action profiles lead to game 2. Replacing $A$ with cooperation ($C$) and $B$ with defection ($D$), the game transition matrix is Eq. SI.77. In Section 2, we have shown that the evolutionary process is insensitive to the initial condition. Inserting Eq. SI.77 into SI.112, we have Eq. SI.111 with $\xi=\left(6k^{4}-10k^{3}+3k^{2}+6k+2\right)/\left(12k^{3}\right)$ for local transitions. Such a transition pattern is a case of $\Omega$. Inserting Eq. SI.77 into SI.114, we have Eq. SI.111 with $\xi=\left(k-1\right)/2$ for global transitions. Eq. SI.111 corresponds to Eq. 1 in the main text. For pairwise-comparison updating, analogously, we have

[TABLE]

where $\xi=\left(10k^{2}-4k+1\right)/\left(24k^{2}-12k\right)$ for local game transitions and $\xi=1/2$ for global game transitions. Eq. SI.115 corresponds to Eq. 2 in the main text.

With the same game transition pattern, in the competition between two stochastic strategies, i.e. $s_{p}$ and $s_{q}$ ($p>q$), from Eqs. SI.100 and SI.102, we have

[TABLE]

under death-birth updating and

[TABLE]

under birth-death or pairwise-comparison updating. Eq. SI.116 shows that for a large value of $p$, game transitions lower the threshold for $\rho_{s_{p}}>\rho_{s_{q}}$ relative to playing a fixed game. Eq. SI.117 shows that game transitions make it possible for cooperative stochastic strategies to be favored over less cooperative stochastic strategies under birth-death or pairwise-comparison updating, which has not been observed when players play a fixed game.

We next consider a game transition pattern under which the evolutionary outcome relies on the initial condition. The game transition matrices are shown in Eq. SI.74. M in Eq. SI.73 is given by

[TABLE]

Switching a few row entries and column entries, we have

[TABLE]

By analyzing $\tilde{\textbf{M}}$, we get $E_{AA}^{(1)},E_{AB}^{(1)},E_{BB}^{(1)}$ belonging to one closed communicating class, and $E_{AA}^{(2)},E_{AB}^{(2)},E_{BB}^{(2)}$ belonging to the other class. Thus, the original system can be reduced into two subsystems, one consisting of $E_{AA}^{(1)},E_{AB}^{(1)},E_{BB}^{(1)}$ and the other consisting of $E_{AA}^{(2)},E_{AB}^{(2)},E_{BB}^{(2)}$. Here, we deal with the subsystem consisting of $E_{AA}^{(1)},E_{AB}^{(1)},E_{BB}^{(1)}$. Substituting $\textbf{P}^{(2)}$, $\textbf{P}^{(1)}$, and $\textbf{P}^{(0)}$ into Eq. SI.42, we have $\dot{\textbf{v}}=\bar{\textbf{A}}\textbf{v}$. $\bar{\textbf{A}}$’s eigenvalues are $\lambda_{1}=0,\lambda_{2}=-(k-2)/(kN),\lambda_{3}=-(2k-2)/(kN)$, where $\lambda_{2}$ and $\lambda_{3}$ are negative for $k>2$. Decomposing $\bar{\textbf{A}}$ into $\bar{\textbf{A}}=\textbf{VDV}^{-1}$, where

[TABLE]

we can get $p_{AA}^{(1)}(t)$, $p_{AB}^{(1)}(t)$ and $p_{BB}^{(1)}(t)$ as a function of time $t$, given by

[TABLE]

where $p_{AA}^{(1)}(t_{0}),p_{AB}^{(1)}(t_{0}),p_{BB}^{(1)}(t_{0})$ are the initial values of $p_{AA}^{(1)},p_{AB}^{(1)},p_{BB}^{(1)}$. When $t$ approaches infinity, we have

[TABLE]

Note that $p_{AA}^{(1)}(t_{0})+2p_{AB}^{(1)}(t_{0})+p_{BB}^{(1)}(t_{0})$ is the initial frequency of game 1, denoted by $p^{(1)}$. Then, we have $p_{AA}^{(1)}$, $p_{AB}^{(1)}$, and $p_{BB}^{(1)}$ as functions of $p_{A}$ and $p^{(1)}$. Analogously, $p_{AA}^{(2)}$, $p_{AB}^{(2)}$, and $p_{BB}^{(2)}$ are functions of $p_{A}$ and $p^{(2)}$. Substituting $p_{AA}^{(1)}$, $p_{AB}^{(1)}$, $p_{BB}^{(1)}$ and $p_{AA}^{(2)}$, $p_{AB}^{(2)}$, $p_{BB}^{(2)}$ into Eqs. SI.44a-SI.44d, we can reduce Eq. SI.53 to

[TABLE]

For donation games, we have

[TABLE]

$\xi=-p^{(2)}$ shows that the evolutionary outcome is sensitive to the initial fractions of various games.

Based on Eqs. SI.86 and SI.87, we can perform the analogous study under global game transitions. With the game transition pattern given in Eq. SI.74, we have $\xi=-p^{(2)}$.

SI.4.4. Evolutionary dynamics with probabilistic game transitions among three states ($n=3$)

All examples that we examine in the main text exhibit a deterministic game transition. That is, the game to be played in the next time step is not probabilistic. In this section, we present an example with stochastic game transitions among three states. Game 1 is the most valuable and game 3 is the least valuable, i.e., $b_{1}>b_{2}>b_{3}$. The game transition matrices are given by

[TABLE]

Mutual cooperation (mutual defection) is prone to yield game $1$ (game $3$) and unilateral cooperation/defection a moderately-valuable game 2. The game transition occurs with a probability $p$ and players play the old game with a probability $1-p$. For $p=0$, by virtue of the approach in Section SI.2, we know that the evolutionary outcome relies on the initial condition. We can refer to the example in Section 4SI.4.3 to derive the evolutionary outcome. For $p>0$, the evolutionary outcome is independent of the initial condition. Note that $p=1$ corresponds to the deterministic case. Under death-birth updating, the general rule for cooperation to be favored over defection is

[TABLE]

For local game transitions, we have

[TABLE]

and

[TABLE]

Here, we analyze how the probabilistic transition measured by $p$ affects the critical benefit-to-cost ratio $\left(b_{1}/c\right)^{*}$. Analyzing $\left(b_{1}/c\right)^{*}=k-\xi_{2}\Delta b_{12}/c-\xi_{3}\Delta b_{13}/c$, we get a threshold

[TABLE]

If $\Delta b_{12}/\Delta b_{13}$ is lower than $\left(\Delta b_{12}/\Delta b_{13}\right)^{*}$, $\left(b_{1}/c\right)^{*}$ decreases monotonically in $p$. If $\Delta b_{12}/\Delta b_{13}$ is larger than $\left(\Delta b_{12}/\Delta b_{13}\right)^{*}$, $\left(b_{1}/c\right)^{*}$ is a non-monotonic function of $p$. Specifically, as $p$ increases starting from a value slightly larger than [math], $\left(b_{1}/c\right)^{*}$ decreases first and then increases. The optimal transition probability $p$ for collective cooperation is

[TABLE]

where

[TABLE]

Depending on the variations in different games, probabilistic game transitions can strengthen the promotive effects of game transitions on the evolution of cooperation in a few cases, whereas weaken them in other cases. The conclusion holds under other updating rules like imitation and pairwise-comparison updating.

For global game transitions, the related parameters are $\xi_{2}=(k-1)/2$ and $\xi_{3}=0$. In this case, probabilistic transitions do not alter the effects of game transitions on the evolution of cooperation.