Control of learning in anti-coordination network games

TL;DR

This paper investigates how to control heterogeneous players in anti-coordination network games to maximize anti-coordination, proposing efficient policies based on vertex cover and cascade potential to reduce control effort.

Contribution

It formulates the control problem as an optimization on anti-coordination, linking it to vertex cover, and introduces cascade-aware policies for improved control efficiency.

Findings

Vertex cover based policy provides a feasible control strategy.

Cascade potential can be exploited for more efficient control.

Cascade-based algorithms significantly reduce control effort.

Abstract

We consider control of heterogeneous players repeatedly playing an anti-coordination network game. In an anti-coordination game, each player has an incentive to differentiate its action from its neighbors. At each round of play, players take actions according to a learning algorithm that mimics the iterated elimination of strictly dominated strategies. We show that the learning dynamics may fail to reach anti-coordination in certain scenarios. We formulate an optimization problem with the objective to reach maximum anti-coordination while minimizing the number of players to control. We consider both static and dynamic control policy formulations. Relating the problem to a minimum vertex cover problem on bipartite networks, we develop a feasible dynamic policy that is efficient to compute. Solving for optimal policies on benchmark networks show that the vertex cover based policy can be a loose upper bound when there is a potential to make use of cascades caused by the learning dynamics of uncontrolled players. We propose an algorithm that finds feasible, though possibly suboptimal, policies by sequentially adding players to control considering their cascade potential. Numerical experiments on random networks show the cascade-based algorithm can lower the control effort significantly compared to simpler control schemes.

Equations

$$u_i(a_i,$$

$$a_i^k$$

$$\Phi (a^0)\equiv (a^0,a^1,\dots )$$

$${\Phi }_k(a^0)\equiv a^k$$

$$u_i(a_i^{NE},a_{{\mathcal{N}}_i}^{\text{NE}})\ge u_i(a_i,a_{{\mathcal{N}}_i}^{\text{NE}})a_i\in [0,1],i\in \mathcal{N}$$

$$B{R}_i(a_{{\mathcal{N}}_i})$$

$$=1\left(1>(c_0(1-s_i)+c_1{s}_i)\sum _{j\in {\mathcal{N}}_i}{a}_j\right)$$

$$a_i^{NE}=B{R}_i(a_{{\mathcal{N}}_i}^{\text{NE}})\text{for all }i\in \mathcal{N}.$$

$$u_i(a_i^{\prime },a_{{\mathcal{N}}_i})>u_i(a_i,a_{{\mathcal{N}}_i})\text{for all }{a}_{{\mathcal{N}}_i}\in {\mathcal{A}}_{{\mathcal{N}}_i}$$

$$A_i^k=\{a_i\in$$

$$a_i\text{ is not strictly dominated by any }{a}_{{\mathcal{N}}_i}\in A_{{\mathcal{N}}_i}^{k-1}\}$$

$$\Phi (y^0,\mathcal{X},\Delta ):=(a^0,a^1,\dots )$$

$$a_i^t$$

$$y^{t+1}$$

$$\underset{\begin{array}{c}{\mathcal{X}}_s\subset \mathcal{N}\\ {\delta }_i\in \{0,1\},i\in {\mathcal{X}}_s\end{array}}{\min }$$

$$a_i^n+a_j^n\le 1\text{for all }(i,j)\in {\mathcal{E}}_B$$

$$a_i^n\in \{0,1\}\text{for all }i\in \mathcal{N}$$

$$\mathcal{X}=\{{\mathcal{X}}_s,{\mathcal{X}}_s,\dots \}$$

$${\delta }_i^t={\delta }_i\in \{0,1\}\text{for all }i\in {\mathcal{X}}_s,\text{for all }t\ge 0$$

$$(a^0,a^1,\dots )=\Phi (\overrightarrow{ϵ},\mathcal{X},\Delta ),\text{ where }\overrightarrow{ϵ}=ϵ1_n$$

$$\underset{\mathcal{X},\Delta }{\min }{J}_d(\mathcal{X},\Delta )$$

$$a_i^t+a_j^t\le 1\text{for all }(i,j)\in {\mathcal{E}}_B,t\ge n$$

$$a_i^t\in \{0,1\}\text{for all }i\in \mathcal{N},t\ge n$$

$$(a^0,a^1,\dots )=\Phi (\overrightarrow{ϵ},\mathcal{X},\Delta )$$

$$\underset{\mathbf{x},\mathbf{\rho }\ge 0}{\min }$$

$$x_i+x_j\ge 1\text{for all }(i,j)\in {\mathcal{E}}^n$$

$$x_i\in \{0,1\}\text{for all }i\in {\mathcal{N}}^n$$

$$c_i\sum _{j\in {\mathcal{N}}_k}(\lceil a_j^n\rceil -x_j)\ge 1-{\rho }_i\text{for all }k\notin {\mathcal{N}}^n$$

$${\mathcal{X}}^n=\{i\in {\mathcal{N}}^n:x_i^{\ast }=1\}\bigcup \{i\in {\mathcal{N}}^n:x_i^{\ast }=0,a_i^n=ϵ\},$$

$${\delta }_i^n=\{\begin{array}{ll}0& \text{for all }i\in \{i\in {\mathcal{N}}^n:x_i^{\ast }=1\}\\ 1& \text{for all }i\in \{i\in {\mathcal{N}}^n:x_i^{\ast }=0,a_i^n=ϵ\}.\end{array}$$

$${\mathcal{X}}^{n+1}=\{i\in {\mathcal{N}}^n:x_i^{\ast }=1\}\bigcup \{i\notin {\mathcal{N}}^n:{\rho }_i^{\ast }>0\},$$

$${\delta }_i^{n+1}=0\text{for all }i\in {\mathcal{X}}^{n+1}.$$

$$a_i^n=\{\begin{array}{ll}1-x_i^{\ast }& \text{if }i\in {\mathcal{N}}^n\\ 0& \text{o.w.}\end{array}$$

$${\mathcal{N}}^n$$

$$\cup \{i\in \mathcal{N}:y_i^n=ϵ,x_i^{\ast }=1\}$$

$$\cup \{i\in \mathcal{N}:y_i^n=ϵ,x_i^{\ast }=0\},$$

$$\mathcal{N}\setminus$$

Full text

I Introduction

Developing methods that achieve globally optimal behavior while conforming with the computational and informational limitations of the players is of interest given the ubiquity of noncooperative interactions that arise among actors in networked systems, e.g., epidemics [1], energy [2, 3], security [4], communication [5] or autonomous systems [6]. Game theoretic learning algorithms are tractable decentralized models for non-cooperative decision-making in networked systems. These algorithms accounting for local information access [7] and coupled action spaces [8] guarantee convergence to individually rational behavior, i.e., a Nash equilibrium action, in certain classes of games, e.g., aggregative [9, 10], potential [11], convex [12]. However, a Nash equilibrium, while being optimal from the perspective of selfish individuals, can be inefficient and undesired at the system level. A canonical example of this is the tragedy of the commons which describes the phenomenon of selfish learning behavior leading to the worst possible outcome for the entire population [13]. Given the possibility of emergence of undesired outcomes, there is a need to develop incentive mechanisms in order to achieve system-wide desired outcomes.

The major challenge in attaining globally desired outcomes in networked systems is that individuals are selfish, heterogeneous, and their actions are coupled while the centralized incentive resources are costly. In this paper, we formulate this challenge as the control of decentralized learning dynamics. That is, players selfishly follow some game-theoretic learning dynamics while a centralized authority aims to direct the emergent behavior toward a desired outcome. The two main issues we address with this formulation are the selection of which players to control, and what control policy to implement given the selected players.

This challenge is addressed in the literature by characterizing the inefficiency of Nash equilibria [14], by designing payoffs prior to start of the game to induce efficient Nash equilibria [15], or by developing control mechanisms [16, 17, 18, 19, 20, 21, 22]. Our approach falls into the last category of controlling players to guide the learningdynamics towards desirable outcomes. In this category, [16, 17] show a public advertising scheme improves the efficiency of the emergent outcome for players that act according to best-response dynamics and occasionally listen to the advertised behavior. An alternative model designs dynamic control incentives that affect every players’ payoffs to which players best respond [18]. Minimum cost uniform and targeted reward policies that induce complete coordination among players that act according to best-response dynamics in a network coordination game are developed in [19]. When the goal is to minimize efficiency, [22] studies malicious attacks that strategically perturb player learning dynamics in a network coordination game. In a combined effort to select which players to control and also how to control them, we depart from these studies by characterizing control schemes when the underlying interaction between players is an anti-coordination game.

In this paper, we focus on controlling a subset of the players in a network anti-coordination game with the aim to promote maximum anti-coordination. Players belong to one of two possible types. We assume there is a preferred selfish action for each player in the absence of any neighboring players in the network. That is, a player’s payoff decreases as more of its neighbors belonging to the opposite type take the preferred action (Section II). Such payoff dependencies can be used to model individual behavior during the spread of an epidemic in a population where the individual types are healthy and sick, the actions represent the level of precautionary measures taken, and payoffs capture the risk of disease transmission to healthy from sick individuals with the preferred action being not taking any measures [23]. Other examples include modeling individual opinions in a politically polarized environment where players would like to differentiate their actions from players in opposing views [24], or modeling two competing species in an environment [25]. In these games, two neighboring players anti-coordinate when at least one of the players do not take the selfish action, e.g., one player in each link takes a precautionary measure during an epidemic. Maximum anti-coordination is achieved when there does not exist any link with failed anti-coordination.

We assume players follow learning dynamics based on a decentralized process of iterated elimination of strictly dominated actions in the absence of any control [26] (Section III). In the learning process, players eliminate all actions that cannot be in a rational action profile in finite time (Theorem 1). This implies convergence to the unique Nash equilibrium in dominance solvable games.

We formulate the minimum player control for anti-coordination problem (MPCAC) as a mixed integer program where the decision variables include which players to control, and how to control them (Section V). We consider static and dynamic control policies. In static MPCAC, a fixed subset of players’ decisions are controlled for the entire learning horizon. In dynamic MPCAC, the control policy can temporally influence player decisions. We find a feasible policy that upper bounds the optimal dynamic MPCAC control policy by solving a minimum cardinality vertex covering problem on a reduced bipartite network (Theorem 2). This feasible policy can be obtained by solving a linear program, and hence is computationally feasible to compute.

In general, the vertex cover-based control policy is sub-optimal if there exists a way to cause cascades of anti-coordination among multi-hop neighbors via the learning dynamics by only controlling a few players. We present optimal policies on benchmark networks of arbitrary size for every possible payoff constant values (Section VI). Some of the optimal policies exhibit use of cascades to achieve anti-coordination which exemplify the sub-optimality of the vertex cover-based control policy. Based on the cascade inducing policies, we also propose a greedy algorithm that finds a feasible solution by sequentially selecting the player to control with the highest potential for inducing a cascade of anti-coordination (Section VII). We compare the greedy algorithm with the vertex cover-based control policy, among several other variants, in numerical experiments on random bipartite networks (Section VIII).

II Anti-coordination network games

A game consists of a population of players $i\in{\mathcal{N}}:=\{1,\dots,n\}$ that take action $a_{i}\in{\mathcal{A}}_{i}$ in order to maximize their utility function $u_{i}(\cdot)$. We assume each player is in one of two possible types $s_{i}\in\{0,1\}$. The population is divided into two disjoint sets ${\mathcal{S}}_{0}:=\{i\in{\mathcal{N}}:s_{i}=0\}$ and ${\mathcal{S}}_{1}:=\{i\in{\mathcal{N}}:s_{i}=1\}$. Only the actions of neighbors that have the opposite type can affect a player’s utility function. For instance, if a player is type 0, then its utility depends on actions of its neighbors in ${\mathcal{S}}_{1}$. We can capture the payoff dependency of players using a bipartite graph ${\mathcal{G}}_{B}=({\mathcal{S}}_{0},{\mathcal{S}}_{1},{\mathcal{E}}_{B})$—see Figure 1. We define the neighborhood of player $i$ as ${\mathcal{N}}_{i}:=\{j\in{\mathcal{N}}:(i,j)\in{\mathcal{E}}_{B}\}$. Then the utility function is defined as $u_{i}:{\mathcal{A}}_{i}\times{\mathcal{A}}_{{\mathcal{N}}_{i}}\to{\mathbb{R}}$ where ${\mathcal{A}}_{{\mathcal{N}}_{i}}=\prod_{j\in{\mathcal{N}}_{i}}{\mathcal{A}}_{j}$. Given the bipartite network ${\mathcal{G}}_{B}$, the network game with binary types can be represented by the tuple $\Gamma=\{{\mathcal{N}},{\mathcal{A}},{\mathcal{G}}_{B},\{u_{i}\}_{i\in{\mathcal{N}}}\}$.

The premise of an anti-coordination game is that a player benefits if its opponents yield. Similarly, in an anti-coordination network game, a player benefits if its neighbors in the opposing type yield. We assume player $i$ can take actions between zero and one, i.e., ${\mathcal{A}}_{i}=[0,1]$ for all $i\in{\mathcal{N}}$. The following utility function,

[TABLE]

with $c_{0}\in(0,1)$ and $c_{1}\in(0,1)$ as constants, captures the preferences of players to differentiate their actions from their neighbors. Here, $a_{{\mathcal{N}}_{i}}$ are the actions of player $i$’s neighbors. Action $a_{i}=1$ maximizes the utility if the term inside the parentheses is positive. Otherwise, action $a_{i}=0$ maximizes the utility. The constant 1 inside the parentheses means that the preferred action is 1 regardless of the type of the player $i$. The term that is subtracted from one captures the decrease in the preference of the player to choose action 1. That is, as $i$’s neighbors increase their action, the benefit of $i$ from choosing the preferred action decreases. This decrease depends on the type of the player. If the player’s type is 0 (1), i.e., $s_{i}=0$ $(=1)$, then the decrease is proportional to $c_{0}$ $(c_{1})$.

Below we provide examples for the anti-coordination network game $\Gamma$ with payoffs as in (1).

Example 1** **(Disease spread on networks)

Players want to avoid disease transmission [23]. Each player is either healthy ($s_{i}=0$) or sick ($s_{i}=1$). The network ${\mathcal{G}}_{B}$ is a contact network with each edge representing a chance of disease transmission between a healthy and a sick player. The action space captures the social distancing level of a player with action $a_{i}=0$ representing self-isolation and action $a_{i}=1$ representing resuming normal activity. Actions between 0 and 1 represent different levels of disease prevention measures, e.g., covering cough, or washing hands often. Resuming normal activity is the preferred action. However, if both players at the two ends of an edge take action 1, then there is a chance of disease transmission. Accordingly, the constant $c_{0}$ captures a healthy player’s sensitivity for avoiding a risky interaction. The constant $c_{1}$ captures a sick player’s sensitivity to avoid transmitting the disease to one of its healthy neighbors.

Example 2** **(Political polarization)

Players want to differentiate their actions from those with opposing beliefs [24]. The network represents the social interactions among players in opposing beliefs (${\mathcal{S}}_{0}$ and ${\mathcal{S}}_{1}$). Action 1 represents a monetary choice or support for a cause that is individually desirable in the absence of partisanship. A player’s tendency to take the preferred action (action 1) reduces as it has more neighbors that take action 1. That is, a player can opt-out from individual benefits or societal impact to express partisan preferences. Constants $c_{0}$ and $c_{1}$ capture the inclination of players in beliefs 0 and 1 to differentiate themselves from the players in the opposing belief, respectively.

Example 3** **(Hawk-Dove network game)

Two competing species (${\mathcal{S}}_{0}$ and ${\mathcal{S}}_{1}$) face-off in an ecological environment. At each interaction players decide to be hawkish ($a_{i}=1$) or dovish ($a_{i}=0$). A hawk move gets the highest reward if its neighboring competitors play dove. If both interacting players play dove, they miss the opportunity to overcome their competitor. If both interacting players are hawkish, they challenge each other and face costs. The constants $c_{0}$ and $c_{1}$ represent the costs species 0 and 1 incur, respectively, when they act hawkish against a hawkish competitor.

III Decentralized learning dynamics

Players repeatedly play the anti-coordination network game $\{{\mathcal{N}},{\mathcal{A}},{\mathcal{G}}_{B},\{u_{i}\}_{i\in{\mathcal{N}}}\}$ taking actions $a_{i}^{k}\in{\mathcal{A}}_{i}$ at each stage $k$. We assume each player knows its own type. Players take actions according to a decentralized algorithm details described in Algorithm 1.

Algorithm 1 starts with each player selecting an arbitrary action $\epsilon\in(0,1)$ not equal to zero or one. Player $i$ checks the worst and best possible outcomes in equations (2)-(3), respectively. In (2), player $i$ checks whether it is preferable to select $a_{i}=1$ according to its utility (1) even when its undecided neighbors ($\{j\in{\mathcal{N}}_{i}:a_{j,k-1}=\epsilon\}$) end up taking action 1. That is, the ceiling operator makes a worst case scenario assumption (all undecided neighbors take action 1) and evaluates its utility from action 1. In (3), player $i$ checks whether it is preferable to select $a_{i}=0$ according to its utility (1) when its undecided neighbors ($\{j\in{\mathcal{N}}_{i}:a_{j,k-1}=\epsilon\}$) end up taking action 0. That is, the floor operator makes a best case scenario assumption (all undecided neighbors taking action 0) and evaluates $a_{i}=0$ according to (1). Note that the ceil and floor operators do not affect the decided neighbors of a player, i.e., neighboring players whose previous action are 0 or 1. If (2)-(3) do not hold at step $k$, player $i$ remains “undecided”, i.e., $a_{i}^{k}=\epsilon$.

Algorithm 1** **(Local learning algorithm)

Initialize:* $a_{i}^{0}=\epsilon$ $\text{for all\ }i\in{\mathcal{N}}$

for $k=1,2,\dots$

Observe $a_{{\mathcal{N}}_{i}}^{k-1}$

[TABLE]

**end

end****

The local algorithm takes as input an initial action profile $a^{0}$, and outputs an infinite sequence of action profiles. We denote the mapping of Algorithm 1 as

[TABLE]

where $a^{t}:=(a_{1}^{t},\ldots,a_{n}^{t})$, We will use the notation

[TABLE]

to denote the resulting action profile after $k$ iterations of the local algorithm.

Figure 2 shows the iterations of Algorithm 1 on a 7 player network for different payoff constants. We observe that depending on the payoff constants $c_{0},c_{1}$, the algorithm might yield an action profile where all players are decided, all remain undecided, or some are decided and some remain undecided. In all cases, the algorithm converges in at most $k=3$ steps. Indeed, the algorithm will only require at most $n$ updates to converge where we recall $n$ as the number of players. Additional number of iterates is not needed because once a player decides on action [math] or $1$, they do not revert. Further, if no player updates at a given step, then it implies there cannot be any updates in the future steps. This means some players can remain undecided. We present the convergence properties of Algorithm 1 in the following section.

IV Convergence of the local learning algorithm

IV-A Game theoretic preliminaries

We define the rational action by the Nash equilibrium solution concept. A Nash equilibrium (NE) action profile $a^{NE}\in[0,1]^{n}$ is such that no individual has a preferable deviation from its action, that is,

[TABLE]

where $a_{{\mathcal{N}}_{i}}^{\text{NE}}:=\{a_{j}^{\text{NE}}:j\in{\mathcal{N}}_{i}\}$. In other words, the individuals respond to the NE actions of other individuals to maximize their payoffs. For a given neighbor action profile $a_{{\mathcal{N}}_{i}}$ we have the best response of individual $i$ as follows,

[TABLE]

where ${\mathbf{1}}(\cdot)$ is an indicator function. Since the payoffs are linear in self-actions, the actions that maximize the payoffs are in the extremes—$a_{i}=1$ or $a_{i}=0$—depending on the types and actions of their neighbors. We can equivalently represent the NE definition in (7) by using the best response definition,

[TABLE]

The notion of strictly dominated is defined as follows.

Definition 1** **(Strictly dominated action)

An action $a_{i}\in[0,1]$ is strictly dominated if and only if there exists an action $a_{i}^{\prime}\in[0,1]$ such that

[TABLE]

If an action $a_{i}$ is strictly dominated then there exists a more preferable action $a_{i}^{\prime}$ for any circumstance. It is clear that if an action is strictly dominated then it cannot be a rational action from (10).

In a game we can iteratively remove the strictly dominated actions, this process is called the iterated elimination of strictly dominated strategies and is defined below.

Definition 2** **(Iterated elimination)

Set the initial set of actions $A_{i}^{0}=[0,1]$ for all $i$, and for any $k\in{\mathbb{N}}$ let

[TABLE]

We denote the set of player $i$’s actions that survive the iterated elimination by $A_{i}^{\infty}:=\bigcap_{k=0}^{\infty}A_{i}^{k}$. When $A_{i}^{\infty}$ has a single element, we say $A_{i}^{\infty}$ is a singleton. If $A_{i}^{\infty}$ is a singleton for all $i\in{\mathcal{N}}$, then the game is dominance solvable, and has a unique Nash equilibrium given by the action profile that survives the iterated elimination process.

IV-B Convergence

The following theorem states that Algorithm 1 eliminates all strictly dominated actions in finite number of steps.

Theorem 1

Algorithm 1 converges in at most $n$ iterations, that is, no player changes its action after the $n$th update. At the end of $n$ iterations, if all players are decided, i.e., $a_{i}^{n}=\{0,1\}$ for all $i\in{\mathcal{N}}$, then all the other actions ${\mathcal{A}}_{i}\setminus a_{i}^{n}$ are strictly dominated and the resultant action profile is a Nash equilibrium. Otherwise, if a player $i\in{\mathcal{N}}$ is undecided, i.e., $a_{i}^{n}=\epsilon$, then there does not exist $a_{i}\in{\mathcal{A}}_{i}$ that can be strictly dominated.

The proof of the theorem is given in the appendix. It relies on showing that Algorithm 1 is a decentralized version of the iterated elimination of strictly dominated actions as given by Definition 2. The intuition for $n$ step convergence is that at each step at least one player needs to eliminate its action using (2) or (3). If no player updates at a time step, the players stop updating because no new eliminations are triggered from then on. Therefore, there could at most be $n$ iterations to rule out $n$ players one player at a time.

V Maximum anti-coordination problem

We say the edge between players $(i,j)\in{\mathcal{E}}_{B}$ is inactive if $a_{i}a_{j}=0$, with $a_{i},a_{j}\in\{0,1\}$. We have maximum anti-coordination when all edges are inactive, that is, $\sum_{(i,j)\in{\mathcal{E}}_{B}}a_{i}a_{j}=0$ with $a_{i},a_{j}\in\{0,1\}$. The learning dynamics (Algorithm 1) does not guarantee that the resulting action profile satisfies maximum anti-coordination. Hence, to ensure maximum anti-coordination, it is necessary to externally control players’ decisions. To this end, we formulate the minimum player control for maximum anti-coordination (MPCAC) optimization problem.

Before we define the optimization problems, we define a controlled action profile trajectory in lieu of Algorithm 1. A control profile is an infinite sequence of subsets of players: ${\mathcal{X}}=\{{\mathcal{X}}^{0},{\mathcal{X}}^{1},\ldots\}$ with ${\mathcal{X}}^{t}\subseteq{\mathcal{N}}$ $\text{for all\ }t\geq 0$. We say $x_{i}^{t}=1$ if player $i\in{\mathcal{X}}^{t}$, and $x_{i}^{t}=0$ otherwise. Furthermore, we denote $\delta_{i}^{t}\in\{0,1\}$ as the forced action of player $i$ at time $t\geq 0$ if $i\in{\mathcal{X}}^{t}$. For convention, and without loss of generality, we say $\delta_{i}^{t}=0$ if $i\notin{\mathcal{X}}^{t}$. We write $\Delta\equiv(\delta^{0},\delta^{1},\ldots)$ for the sequence of forced action profiles. With the control profile ${\mathcal{X}}$, the resulting action profile trajectory, with intial action profile $y^{0}$ is written

[TABLE]

where the $a^{t}$ obey the following dynamics for $t=0,1,\ldots$

[TABLE]

Note $\Phi_{1}(a^{t})$ is the uncontrolled action at time $t+1$ by (6) given the controlled action profile $a^{t}$ at time $t$. We will refer to the pair $({\mathcal{X}},\Delta)$ as a control policy. In the first formulation of the maximum anti-coordination optimal control problem, we seek to achieve maximum anti-coordination using as few fixed control players as possible. It is formalized as follows.

Definition 3** **(Static MPCAC)

[TABLE]

The objective in (15) is to minimize the cardinality of the set ${\mathcal{X}}_{s}$, that is, the number of players controlled. The first and second constraints together make sure of maximum anti-coordination at time $n$ when all players must be decided, i.e., they are equivalent to the constraint $\sum_{(i,j)\in{\mathcal{E}}_{B}}a_{i}a_{j}=0$ with $a_{i}^{n}\in\{0,1\}$ for all $i\in{\mathcal{N}}$. The third and fourth constraints define the controlled players and their actions for all times. The third constraint selects forced actions in the control set. The last constraint states the controlled learning dynamics (13) that govern players’ decision-making.

The optimization problem (15) runs for $n$ time steps to allow for convergence of Algorithm 1 because if the algorithm dynamics are going to generate an anti-coordinating action pair, we would want to make use of it instead of incurring the cost for controlling a player. Selecting a player to the controlled set and changing its action via $\delta_{i}$ does not increase the convergence time of the algorithm, hence we stop the optimization after $n$ steps. Indeed, if we control a set of players ${\mathcal{X}}_{s}$ at $t=0$, by Theorem 1 the algorithm converges in $n-|{\mathcal{X}}_{s}|$ steps. Hence, the optimization horizon of $n$ steps is sufficient to make full use of learning dynamics.

The following result uses the convergence in finite time of the iterated elimination process to show that any static policy that satisfies the maximum anti-coordination constraint at time $n$ will continue to satisfy it for $t>n$.

Lemma 1

If $\Pi_{s}=({\mathcal{X}}_{s},\Delta_{s})$ is a static control policy such that $a_{i}^{n}+a_{j}^{n}\leq 1$ for all $(i,j)\in{\mathcal{E}}_{B}$, then no player will change its decision, that is, $a_{i}^{t}+a_{j}^{t}\leq 1$ for all $(i,j)\in{\mathcal{E}}_{B}$, $t>n$.

**Proof : **A control profile ${\mathcal{X}}_{s}$ with actions $\delta_{i}=\{0,1\}$ is feasible if all players decide by time $n$. Given a feasible static control policy $({\mathcal{X}}_{s},\Delta)$, define the game $\Gamma^{\prime}$ among players ${\mathcal{N}}\setminus{\mathcal{X}}_{s}$ where players connected to ${\mathcal{X}}_{s}$ have a set of decided neighbors according to forced action profile $\Delta$. The game $\Gamma^{\prime}$ must be dominance solvable so that Algorithm 1 converges by time $n$. Thus if we continue to apply the forced actions $\Delta$, no player would change its decision after time $n$. ∎

In static MPCAC, we decide on players to control at the beginning and set their actions for the entire horizon. The objective only accounts for the number of players controlled but not the number of times we control a player. In many situations it may be enough to control a player for a finite time to achieve maximum anti-coordination. For instance, in case (B) in Fig 2, if we set the actions of players 5-7 to 0 for one time step, the remaining players (1-4) will take action 1 by (2). If we stop controlling the players 5-7 in the next time step, they will continue to take action 0 by (3). Hence, the resultant action profile will achieve maximal anti-coordination. Next we formulate the dynamic MPCAC problem that allows for dynamic selection of players to control, and accounts for the number of times we control each player.

Definition 4** **(Dynamic MPCAC)

[TABLE]

The two terms in the penalty function in (22) equally weight the control effort per player before convergence and after convergence to maximum anti-coordination. In dynamic MPCAC, we allow for the set of controlled players and their actions to change at each step. Lemma 1 does not necessarily apply in the dynamic setting. Hence, we explicitly require that the maximum anti-coordination is maintained for all times after $n$ in the first two constraints. The last constraint specifies the controlled learning dynamics (13).

The following result shows that an optimal policy for dynamic MPCAC should at least be as good as an optimal policy for static MPCAC.

Lemma 2

Let $\Pi_{s}=({\mathcal{X}}_{s}^{*},\delta^{*})$ and $\Pi_{d}=({\mathcal{X}}_{*},\Delta_{*})$ be optimal policies for static and dynamic MPCAC, respectively. Then $J_{d}(\Pi_{d})\leq 2J_{s}(\Pi_{s})=2|{\mathcal{X}}_{s}^{*}|$.

**Proof : **Suppose there exists an optimal dynamic policy $\Pi_{d}$ such that $J_{d}(\Pi_{d})>2|{\mathcal{X}}_{s}^{*}|$. Then we can implement $(\Pi_{s})_{t\geq 0}$ to achieve a cost of $2|{\mathcal{X}}_{s}^{*}|$ in dynamic MPCAC, where the first and second terms in the objective 22 will be $|{\mathcal{X}}_{s}^{*}|$. By Lemma 1, $\Pi_{s}$ will satisfy the constraints in dynamic MPCAC which means it is also a feasible solution for the dynamic MPCAC. Hence, $\Pi_{d}$ cannot be optimal. ∎

This result is expected when we observe that any static policy that is feasible for the static MPCAC is also feasible for the dynamic MPCAC by Lemma 1. Hence, we can always use the optimal solution for static MPCAC to upper bound the penalty in the dynamic MPCAC. In fact, if the static MPCAC solution reaches an action profile that is an equilibrium of the game by time $n$, then the optimal solution is upper bounded by $|{\mathcal{X}}_{s}^{*}|$. That is, we do not need to make any control efforts to remain at the maximum anti-coordination action profile because the action profile is also an equilibrium of the game.

The first constraint of dynamic MPCAC requires maximum anti-coordination after time $n$, whether or not control actions are used to keep it in equilibrium. This together with the penalization of control efforts for all times after $n$ gives preference to control policy solutions, when feasible, that leverage the controlled dynamics $\Phi$ in (13) in order to achieve maximum anti-coordination. The following result supports this intuition for dominance solvable games.

Lemma 3

If the game is dominance solvable, the optimal policy for dynamic MPCAC ($\Pi_{d}=({\mathcal{X}}_{*},\Delta_{*})$) is either ${\mathcal{X}}^{t}_{*}=\emptyset$ for all $t=1,2,\dots$ or ${\mathcal{X}}^{t}_{*}\neq\emptyset$ for $t\geq n$.

The above result proven in the Appendix shows that if Algorithm 1 converges to a single action profile, then either no control effort is required or the control effort after convergence ($t>n$) is non-zero. This result relies on whether the unique Nash equilibrium achieves maximum anti-coordination or not, which then corresponds to an empty control profile or a non-empty control profile for $t>n$, respectively. In general, if the game $\{{\mathcal{N}},{\mathcal{A}},{\mathcal{G}}_{B},\{u_{i}\}_{i\in{\mathcal{N}}}\}$, not necessarily dominance solvable, has Nash equilibria that achieve maximum anti-coordination, we would expect control policies that induce convergence to such Nash equilibria over policies that control players after time $n$ in order to avoid the second term in the objective of dynamic MPCAC. We formalize this intuition in the following lemma (see appendix for the proof).

Lemma 4

If there exists a set of Nash equilibria that achieves maximal anti-coordination, then the optimal solution to dynamic MPCAC will reach an action profile in this set.

Next we relate the dynamic MPCAC problem to a vertex covering problem on a bipartite network. We define the graph with potential dangerous links after Algorithm 1 converges at time $n$ as follows. Let ${\mathcal{G}}^{n}=({\mathcal{N}}^{n},{\mathcal{E}}^{n})$ be the graph with vertices composed of players that remain undecided or take action 1 after Algorithm 1 converges at step $n$, i.e., ${\mathcal{N}}^{n}:=\{j\in{\mathcal{N}}:a_{j}^{n}=\{\epsilon,1\}\}$, and with edges that connect players in ${\mathcal{N}}^{n}$ in ${\mathcal{E}}_{B}$, that is, ${\mathcal{E}}^{n}=\{(i,j)\in{\mathcal{E}}_{B}:i\in{\mathcal{N}}^{n},j\in{\mathcal{N}}^{n}\}$. For example, in Fig. 2 case (A), the network ${\mathcal{G}}^{n}$ has the vertex set ${\mathcal{N}}^{n}=\{1,3,4,5,6,7\}$.

A cardinality vertex cover for the graph ${\mathcal{G}}^{n}=({\mathcal{N}}^{n},{\mathcal{E}}^{n})$ looks for a minimum cardinality subset of vertices ${\mathcal{X}}^{n}\subseteq{\mathcal{N}}^{n}$ such that each edge has at least one endpoint incident at ${\mathcal{X}}^{n}$ [27]. Given the action profile at time $n$, $a^{n}$, we consider the following modified cardinality vertex cover problem with tuning parameter $\lambda\geq 0$,

[TABLE]

where $c_{i}:=c_{0}(1-s_{i})+c_{1}s_{i}$ is player $i$’s payoff constant. If the second term in the objective and the last constraint are excluded, the problem formulation would be the minimum vertex covering problem in the bipartite network ${\mathcal{G}}^{n}$. If we set $\rho_{i}=0$ for players $i\in{\mathcal{N}}\setminus{\mathcal{N}}^{n}$, the last constraint makes sure that we select the players in the vertex covering such that the players in ${\mathcal{N}}\setminus{\mathcal{N}}^{n}$ who are decided on action 0 do not change their actions as a result of the control efforts. In general, the last constraint with $\rho_{i}=0$ can make the vertex covering problem infeasible—see Figure 3 for an example.

Let ${\mathbf{x}}^{*}=(x_{1}^{*},\dots,x_{n}^{*})$, and $\boldsymbol{\rho}=(\rho_{1}^{*},\dots,\rho_{n}^{*})$ be an optimal solution to (27). We construct a feasible control policy $\Pi_{v}=\{{\mathcal{X}},\Delta\}$ as follows. Let ${\mathcal{X}}^{t}=\emptyset$ if $t<n$. Define the controlled player set ${\mathcal{X}}^{n}$ and their actions $\Delta^{n}$ at time $n$ based on the optimal solution of the above problem as follows,

[TABLE]

and

[TABLE]

For $t=n+1$, we define the controlled player set ${\mathcal{X}}^{n+1}$ and their actions $\Delta^{n+1}$ as follows,

[TABLE]

and

[TABLE]

For $t\geq n+1$, we let ${\mathcal{X}}^{t}={\mathcal{X}}^{n+1}$ if the action profile $\{a_{i}^{n+1},\delta_{i}^{n+1}\}_{i\in{\mathcal{N}}}$ is not a Nash equilibrium of the game. Otherwise, ${\mathcal{X}}^{t}=\emptyset$ for $t>n$.

Theorem 2

The dynamic control policy $\Pi_{v}=\{{\mathcal{X}},\Delta\}$ with control and action sets at time $n$ given by (31)-(32) and for time $t\geq n+1$ given by (33)-(34) is a feasible dynamic policy for the dynamic MPCAC problem in (22).

**Proof : **Given (31)-(32), the controlled action profile at time $n$ is given by (14),

[TABLE]

Given the definition of ${\mathcal{G}}^{n}$ and $\Delta^{n}$, we have the controlled actions $a^{n}$ satisfy $a_{i}^{n}+a_{j}^{n}\leq 1$ for any $(i,j)\in{\mathcal{E}}^{n}$, since we force at least one player in every link of ${\mathcal{G}}^{n}$ to play action 0. This means that at time $n$ anti-coordination is achieved by $a^{n}$. Further, all players are decided satisfying the second constraint in (22).

Next, we show ${\mathcal{X}}^{n+1},\Delta^{n+1}$ maintains anti-coordination with the controlled action profile $a^{n+1}$.

Consider the following set partition of ${\mathcal{N}}^{n}$,

[TABLE]

where $y^{n}$ is the uncontrolled action profile at time $n$. For the first two set of players in (38), we have $a_{i}^{n+1}=1-x_{i}^{*}$ by (33). The last set of players were controlled to play action 1. At time $n$, they have no neighbors that take action 1 in (32). So they continue to take action 1 using the update (2) even when they are not controlled any longer. Hence, $y_{i}^{n+1}=1-x_{i}^{*}$ for $\{i\in{\mathcal{N}}:a_{i}^{n}=\epsilon,x_{i}^{*}=0\}$. This implies that $a_{i}^{n+1}=1-x_{i}^{*}$ for all $i\in{\mathcal{N}}^{n}$.

Next, consider the following partition of the set of players not belonging to ${\mathcal{N}}^{n}$,

[TABLE]

Note that all the players in the first set in (40) are controlled to play action 0 in (33), that is, $a_{i}^{n+1}=0$ because $\{i\in{\mathcal{N}}\setminus{\mathcal{N}}^{n}:a_{i}^{n}=0,c_{i}\sum_{j\in{\mathcal{N}}_{i}}(\lceil a_{j}^{n}\rceil-x_{j}^{*})<1\}=\{i\notin{\mathcal{N}}^{n}:\rho_{i}^{*}>0\}$ by the last constraint in (27). For the second set in (40), all players continue to select action 0 by (3). This implies that $a_{i}^{n+1}=0$ for $i\notin{\mathcal{N}}^{n}$.

Combining the two arguments, we have $a_{i}^{n+1}=a_{i}^{n}$ where $a_{i}^{n}$ is given in (35). Hence, the controlled action profile at time $n+1$ achieves anti-coordination. Suppose, $a^{n+1}$ is a Nash equilibrium action profile, then $a^{n+1}=\Phi_{1}(a^{n+1})$ and no further control effort is necessary. Otherwise, we have $y^{n+2}=\Phi_{1}(a^{n+1})=\Phi_{1}(a^{n})=y^{n+1}$ where $y^{n+1}$ and $y^{n+2}$ are the uncontrolled action profiles at time $n+1$ and $n+2$, respectively. Since, ${\mathcal{X}}^{t}={\mathcal{X}}^{n+1}$, $\Delta^{t}=\Delta^{n+1}$ for $t\geq n+1$, we have that $a^{n+2}=a^{n+1}$. By induction, the control policy $\Pi_{v}$ satisfies anti-coordination. ∎

The proof relies on showing that a minimum vertex cover will eliminate all possible risky interactions after Algorithm 1 possibly eliminates a subset of them. The optimization formulation in (27) makes sure that there are no changes to the players decided on action 0 by time $n$. The minimum vertex cover solution ${\mathcal{X}}^{n}$ and forced actions $\delta_{i}^{n}=0$ for all $i\in{\mathcal{X}}^{n}$ makes sure that the first two constraints in dynamic MPCAC are satisfied. The policy after time $n$ makes sure that all players continue to take the same actions for time $t>n$ hence satisfying the first two constraints for all $t>n$.

Theorem 2 shows that the MPCAC problems can be upper bounded by solving the minimum vertex cover on the reduced bipartite graph ${\mathcal{G}}^{n}$. Further, if we set the tuning parameter $\lambda=0$, then the integer program in (27) has an exact linear programming relaxation due to total unimodularity of bipartite networks [28, Ch. 3]. Hence, we can obtain a feasible policy efficiently by solving (27) with penalty term $\lambda=0$. Then, including all players that have positive $\rho_{i}$ in the control set ($\{i\notin{\mathcal{N}}^{n}:\rho_{i}^{*}>0\}$). The following corollary presents a scenario in which $\Pi_{v}$ optimal.

Corollary 1

The policy $\Pi_{v}$ defined in Theorem 2 is an optimal policy if all players eliminate their actions in one time step in Algorithm 1.

The proof given in the appendix relies on showing that when all players eliminate their actions, the optimization problem in (27) reduces to solving the minimum cardinality vertex cover for the entire network ${\mathcal{G}}$. This case happens only when $c_{i}$ is smaller than inverse of the maximum degree of the network ($c_{i}|{\mathcal{N}}_{i}|<1$ for all $i\in{\mathcal{N}}$). The control policy $\Pi_{v}$ is an upper bound in general because it does not make use of cascades, i.e., use the learning dynamics to make multiple hop links inactive, by controlling a subset of players. Instead, it waits for the algorithm to eliminate as many active links as possible, and then makes a two time-step control of players. As we show in numerical examples in Section VIII, the policy $\Pi_{v}$ tends to perform well when $c_{0}$ is small and $c_{1}$ is large, or when $c_{0}$ is large and $c_{1}$ is small. In the following section, we present optimal policies for benchmark networks that exemplify the optimal use of cascades to eliminate active links.

VI Optimal solutions for benchmark networks

We consider star, line and ring networks, and provide optimal solutions for both static and dynamic MPCAC problems.