A Differential Game Approach to Decentralized Virus-Resistant Weight Adaptation Policy over Complex Networks

TL;DR

This paper models virus spread in complex networks using a differential game framework, proposing a decentralized weight adaptation policy to mitigate malware propagation and improve network resilience.

Contribution

It introduces a novel differential game approach for decentralized virus mitigation and designs a penalty-based mechanism to align individual actions with social welfare.

Findings

Nash equilibrium structure characterized in the epidemic control game.

Decentralized weight adaptation reduces virus spread effectively.

Penalty scheme improves overall network resilience.

Abstract

Increasing connectivity of communication networks enables large-scale distributed processing over networks and improves the efficiency for information exchange. However, malware and virus can take advantage of the high connectivity to spread over the network and take control of devices and servers for illicit purposes. In this paper, we use an SIS epidemic model to capture the virus spreading process and develop a virus-resistant weight adaptation scheme to mitigate the spreading over the network. We propose a differential game framework to provide a theoretic underpinning for decentralized mitigation in which nodes of the network cannot fully coordinate, and each node determines its own control policy based on local interactions with neighboring nodes. We characterize and examine the structure of the Nash equilibrium, and discuss the inefficiency of the Nash equilibrium in terms of minimizing the total cost of the whole network. A mechanism design through a penalty scheme is proposed to reduce the inefficiency of the Nash equilibrium and allow the decentralized policy to achieve social welfare for the whole network. We corroborate our results using numerical experiments and show that virus-resistance can be achieved by a distributed weight adaptation scheme.

Tables

Table 1. TABLE I: Parameters used for numerical study unless otherwise stated.

Cost functions                       Network Topology                            Spreading
$f_i\left(x_i\right)$	$g_{ij}\left(u_{ij}-w_{ij}^o\right)$	${\alpha }_i$	$d_{ij}$	$N$	$w_{ij}^o$	$⟨k^{out}⟩$	$⟨k^{in}⟩$	${\beta }_i$	${\sigma }_i$	$T$
${\alpha }_i{x}_i$	$\frac{1}{2}{d}_{ij}{\left(u_{ij}-w_{ij}^o\right)}^2$	$1$	$0.2$	$150$	$1$	$7.545$	$7.545$	$0.04$	$0.1$	20

Equations

$$\mathbb{P}(X_i(t+\Delta t)=1|X_i(t)=0,X(t))$$

$$=\sum _{j=1}^N{w}_{ij}{\beta }_j{X}_j(t)\Delta t+o(\Delta t),$$

$$\mathbb{P}(X_i(t+\Delta t)=0|X_i(t)=1,X(t))={\sigma }_i\Delta t+o(\Delta t).$$

$${\dot{x}}_i(t)=(1-x_i(t))\sum _{j=1}^N{w}_{ij}(t){\beta }_j{x}_j(t)-{\sigma }_i{x}_i(t),$$

$$\dot{\mathbf{x}}(t)=G(\mathbf{x}(t),\mathbf{W}(t)),$$

$$J_i=\underset{0}{\overset{T}{\int }}{f}_i(x_i(t))+\sum _{j=1}^N{g}_{ij}(w_{ij}(t)-w_{ij}^o)dt.$$

$$\underset{{\mathbf{w}}_i\in S_i}{\min }$$

$$\text{(refEpDy)},$$

$$J_1({\mathbf{w}}_1^{\ast },{\mathbf{w}}_2^{\ast },...,{\mathbf{w}}_N^{\ast })$$

$$J_2({\mathbf{w}}_1^{\ast },{\mathbf{w}}_2^{\ast },...,{\mathbf{w}}_N^{\ast })$$

$$⋮$$

$$J_N({\mathbf{w}}_1^{\ast },{\mathbf{w}}_2^{\ast },...,{\mathbf{w}}_N^{\ast })$$

$$\underset{{\mathbf{u}}_i\in U_i}{\min }{J}_i=\underset{0}{\overset{T}{\int }}{f}_i(x_i(t))+\sum _{j\in {\mathcal{N}}_{i,o}^{out}}{g}_{ij}(u_{ij}(t)-w_{ij}^o)dt$$

$$\text{s.t. }{\dot{x}}_i(t)=(1-x_i(t))\sum _{j\in {\mathcal{N}}_{i,o}^{out}}{u}_{ij}(t){\beta }_j{x}_j(t)-{\sigma }_i{x}_i(t),$$

$$x_i(0)=x_{i0},i=1,2,...,N,$$

$$w_{ij}^{\ast }(t)=\{\begin{array}{ll}{u}_{ij}^{\ast }(t)& \text{if}j\in {\mathcal{N}}_{i,o}^{out}\\ 0& \text{otherwise}\end{array}$$

$${\dot{x}}_i^{\ast }(t)$$

$$x_i^{\ast }(0)=x_{i0},\forall i\in \mathcal{N},$$

$${\mathbf{u}}_i^{\ast }(t)=\arg \underset{{\mathbf{u}}_i\in U_i}{\min }$$

$$H_i(t,{\mathbf{p}}_i(t),{\mathbf{x}}^{\ast },{\mathbf{u}}_1^{\ast }(t),...,{\mathbf{u}}_{i-1}^{\ast },{\mathbf{u}}_i,{\mathbf{u}}_{i+1}^{\ast }(t),...,{\mathbf{u}}_N^{\ast }(t)),$$

$${\dot{\mathbf{p}}}_i(t)={\Gamma }_i(t,{\mathbf{x}}^{\ast },{\mathbf{u}}_1^{\ast },...,{\mathbf{u}}_N^{\ast }){\mathbf{p}}_i(t)+{\gamma }_i(t),{\mathbf{p}}_i(T)=0,$$

$$H_i(t,{\mathbf{p}}_i,\mathbf{x},{\mathbf{u}}_1,...,{\mathbf{u}}_N)\triangleq f_i(x_i(t))+\sum _{j\in {\mathcal{N}}_{i,o}^{out}}{g}_{ij}(u_{ij}-w_{ij}^o)$$

$$+\sum _{j=1}^N{p}_{ij}\left\{(1-x_j)\sum _{k\in {\mathcal{N}}_{j,o}^{out}}{u}_{jk}{\beta }_k{x}_k(t)+{\sigma }_j{x}_j(t)\right\},$$

$${\Gamma }_{i,mn}=\{\begin{array}{ll}\sum _{j\in {\mathcal{N}}_{m,o}^{out}}{u}_{mj}^{\ast }(t){\beta }_j{x}_j^{\ast }(t)+{\sigma }_m& \text{if}n=m\\ -(1-x_n^{\ast }(t))u_{nm}^{\ast }(t){\beta }_m& \text{if}n\in {\mathcal{N}}_{m,o}^{in}\\ 0& \text{otherwise},\end{array}$$

$$u_{ij}^{\ast }(t)=$$

$$\{\begin{array}{ll}0,& -{\varphi }_{ij}(t)\le g_{ij}^{\prime }(-w_{ij}^o),\\ (g_{ij}^{\prime }{)}^{-1}(-{\varphi }_{ij}(t)),& g_{ij}^{\prime }(-w_{ij}^o)<-{\varphi }_{ij}(t)<g_{ij}^{\prime }(0),\\ w_{ij}^o,& -{\varphi }_{ij}(t)\ge g_{ij}^{\prime }(0),\end{array}$$

$$u_{ij}^{\ast }(t)=\{\begin{array}{ll}0,& {\varphi }_{ij}(t)\ge \frac{g_{ij}(-w_{ij}^o)}{w_{ij}^o},\\ w_{ij}^o,& {\varphi }_{ij}(t)<\frac{g_{ij}(-w_{ij}^o)}{w_{ij}^o},\end{array}$$

$$\displaystyle\min\limits_{\mathbf{u}\in U_{o}}J_{o}=\int\limits_{0}^{T}\sum\limits_{i=1}^{N}f_{i}(x_{i}(t)){\color[rgb]{0,0,1}+}\sum\limits_{i=1}^{N}\sum\limits_{j\in\mathcal{N}_{i,o}^{out}}g_{ij}(u_{ij}(t)-w^{o}_{ij})dt$$

$$s.t.{\dot{x}}_i(t)=(1-x_i(t))\sum _{j\in {\mathcal{N}}_{i,o}^{out}}{u}_{ij}(t){\beta }_j{x}_j(t)-{\sigma }_i{x}_i(t),$$

$$x_i(0)=x_{i0},i=1,2,...,N,$$

$${\dot{x}}_i^o(t)$$

$$\dot{\lambda }(t)$$

$${\mathbf{u}}^o(t)$$

$$H(t,\mathbf{x}(t),\lambda (t),\mathbf{u}(t))$$

$$=$$

$$+$$

$${\int }_0^T$$

$$={\int }_0^T\pi (t,x_i,{\mathbf{x}}_{-i},{\mathbf{u}}_i,{\mathbf{u}}_{-i})-\pi (t,{\hat{x}}_i,{\hat{\mathbf{x}}}_{-i},{\mathbf{v}}_i,{\mathbf{u}}_{-i})dt,$$

$${\hat{J}}_i={\int }_0^T{\hat{l}}_i(t)dt=J_i+{\int }_0^T{c}_i(t)dt,$$

$${\dot{\mathbf{p}}}_i(t)=\Gamma (t){\mathbf{p}}_i(t)+{\hat{\gamma }}_i(t)$$

Full text

I Introduction

The integration of the information and communications technologies into systems upgrades system performance. However, the integration also degrades the security level of the systems and introduces vulnerabilities that undermine the reliability of critical infrastructure. The connectivity and interdependence of cyber networks make the system even more vulnerable due to the existence of the wide-spreading cyber-attacks on networks. It provides opportunities for the sophisticated and stealthy malware and virus to spread over the network. One noteworthy example is the StuxNet attack [1]. In June 2010, certain control systems of a nuclear-enrichment plant in Iran were infected by a carefully crafted computer worm called StuxNet. The worm, spreading through USB devices, intended to breach the implemented cyberprotection schemes and alter both the measurement and actuation signals which caused instabilities and damage the physical plant[2]. More recent examples of wide-spreading cyber-attacks include WannaCry and Petya Ransomware, which have incurred billions of dollars of losses [3].

With an increasing number of wide-spreading cyber-attacks on networks, protection against malware and virus spreading in cyber networks is central to the security of network systems [3]. However, there are many challenges on designing a protection scheme for cyber networks. One challenge is due to the interderdependency between the microscopic individual behaviors and the macroscopic spreading phenomenon. The local interactions over a large network where nodes communicate, share information, and make interdependent decisions, can result in a macroscopic behavior, which will in turn affect the agents’ behaviors. This type of microscopic and macroscopic couplings has been illustrated in Fig. 1. Another challenge arises from the fact that cyber networks are often formed by a large number of self-interested agents or decision-makers. The noncooperation among the agents makes it almost impossible for the system to be coordinated as a whole to defend against wide-spreading cyber-attacks.

To this end, one way to mitigate the malware spreading over large networks is to control the intensity of interactions with neighboring nodes. By adapting the rate of communications or contacts, nodes can reduce the likelihood of infection. This type of mechanism is called weight adaptation as the weights between two nodes of a network capture the intensity of the connectivity [4]. The most fundamental reason that virus and malware can go viral is the inherent property of networks: connectivity. Weight adaptation is a mechanism that hits the nail. Weight adaptation lowers the connectivity which leaves virus and malware no way out. Compared with quarantining and link removal [5], weight adaptation does not need to completely disconnect nodes from others but rather adjust weights to connect more loosely with nodes with a higher likelihood of infection. Instead of fixing the weights for the whole spreading process, in the weight adapation scheme, each agent dynamically updates their weight in response to the state of the neighboring nodes. Weight adaptation is different from changing the infection rate. The infection rate is usually considered to be decided by some interior factors like physiological or immunological states of individual. The weight between two nodes is usually used to describe how strongly two nodes are connected. Changing the weight can be interpreted as an exterior change.

We consider a directed weighted network where the nodes and the edges represent the agents and the connections between the agents respectively. The directed connection between two nodes can be considered as one agent acquiring information/data/packet from another agent. The weight between two agents quantifies the frequency or the volume of communication between two agents [6]. The original weight is pre-designed by multilateral agreement among agents to achieve certain goals or to optimize the system performance when there is no infection. For example, in distributed estimation or learning problems over networks such as [7, 8, 6], one agent needs to communicate with its neighboring agents at a sufficient rate to find the global estimate of the state. The optimal weighting on the edges quantitatively captures the minimum required frequency of contacting neighboring nodes. As illustrated in Fig. 2, when there are wide-spreading virus or cyber attacks, the agent can decrease the likelihood of being infected by reducing their weight with infected neighbors. The agent then restores the connections when the infected neighbors are recovered. Deviation of the weights from the optimal ones introduces cost induced by performance degradation and system inefficiency. Infected agents may not function normally. The agents and the network system will suffer losses. Thus, it is essential to consider the trade-off between malfunction cost caused by infection and inefficiency or performance degradation cost caused by weight deviation.

In this paper, an $N$-person nonzero-sum differential game-based model is proposed to model the virus spreading and the agents’ adaptive response to virus infection. This model captures the non-cooperative behaviors among agents, dynamic properties of spreading process, and the complexity of the local interactions. We characterize the Nash equilibrium (NE) for the game and investigate the network effects under the non-cooperative strategies. We observe that under the open-loop NE, each agent updates his weight based on its own infection level and its out-neighbors infection level as well as the corresponding component of its costate. When the agent’s own infection level is high, it does not care much about the weight of links to infected out-neighbors. When its out-neighbor’s infection level is high, it lowers more weight of the corresponding connection. The corresponding component of each agent’s costate encodes the information about the network structure and the infection of the whole network.

We use a centralized optimal control problem to serve as a benchmark problem to study the efficiency of the decentralized problem. Under centralized policies, the system operator develops optimal weight adaptation scheme to achieve social optimum. Compared with the centralized solution, the open-loop NE solution is not the best from a system point of view since in the game, agents consider only their own cost. Such inefficiency caused by selfish behavior of agents has a significant impact on network and service management. One example is the congestion in traffic network caused by selfish drivers [9]. To address the inefficiency, we propose a dynamic penalty approach by designing a mechanism in which each agent pays for the infection cost of all agents that are reachable to him/her. We show that with this mechanism, the open-loop NE policy achieves the social optimum.

The equilibrium analysis and the mechanism design lead to a distributed algorithm for the network operator and the agents to compute the optimal weight adaptation where each agent only has to know local information. We summarize the principal contributions as follows:

We propose a differential game model to develop a virus-resistant weight adaptation scheme for cyber networks formed by a group of self-interested agents. 2. 2.

We study the structure of the open-loop NE for the differential game over complex networks and show the weight adaptation rule is based on the agents’ and its out-neighbors’ infection level as well as the costate. 3. 3.

We discuss the inefficiency of the NE. A dynamic penalty scheme is proposed to achieve social optimum for the whole network. 4. 4.

An implementable distributed virus resistance algorithm is proposed to compute the NE-based control policy.

Game theory has long been a useful tool to design strategies on network systems for virus resistance purposes [3, 10, 11, 12]. In [10], the authors have proposed a network formation game that balances multiple partially conflicting objectives such as the cost of installing links, the performance of the network and the resistance to virus. In their work, an undirected unweighted static network is formed. Hayel et.al. in [3, 11] have studied large population game with heterogeneous types of individuals. They focus on group behavior of certain type in stead of individual behavior. Besides game theory, other tools such as impulse control[13], optimal control[14], and optimization [15] have been used to design strategies to mitigate malware attacks and virus spreading.

Virus spreading over adaptive networks has first been studied by Gross et.al. in [16]. They investigated adaptive behavior in a homogeneous way where the whole network takes the same adaption. Based on the work on epidemic spreading over time-varying networks [17], optimal control method has been utilized to find the optimal time-varying topology response for the network system in [14]. However, the centralized optimal control method is not practical and lack of incentive. The effect of heterogeneous weight adaptation on virus spreading has been studied by Yun et al. in [18, 19]. In [18], the authors have proposed a weight adaptation rule without taking cost into consideration. The weight adaptation rule is based on the infection level of the whole network.

Vaccination and immunity have been studying for control of virus spreading over decades [13, 15, 20]. But vaccination may not be efficient for some malware and virus due to their fast upgrading and undetectability. Also, getting every individual vaccinated is costly. Quarantining [5] is equivalent to removal of all connections of one agent. Compared with weight adaptation scheme, it is overreacting to disconnect all links since connection with healthy agents cause no harm.

The paper is organized as follows. In Sect. II, preliminaries are given and the $N$-person nonzero-sum differential game framework is introduced. Section III describes the open-loop NE of the differential game and the weight adaptation scheme. Sect. IV studies the efficiency of the NE solution. Comparisons of the differential game-based weight adaptation scheme with the optimal control based scheme and other numerical results are given in Sect. V. Conclusions are contained in Sect. VI.

II Preliminaries and Problem Formulation

In this section, we introduce notations and preliminary results needed in our derivations. Along the way, we describe and develop the problem formulation.

II-A Graph Theory

A weighted, directed graph can be defined by a triple $\mathcal{G}\triangleq(\mathcal{V},\mathcal{E},\mathcal{W})$. $\mathcal{V}\triangleq\{v_{1},v_{2},...,v_{N}\}$ represents a set of $N$ nodes. Define $\mathcal{N}\triangleq\{1,...,N\}$. A set of directed edges is denoted by $\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$. The set of in-neighbors of node $i$ is defined as $\mathcal{N}_{i}^{in}\triangleq\{j|j\in\mathcal{V},(j,i)\in\mathcal{E}\}$. Denote by $|\cdot|$ the cardinality of a set. So, the in-degree of $v_{i}$ is $|\mathcal{N}_{i}^{in}|$. Similarly, the set of out-neighbors of $v_{i}$ is $\mathcal{N}_{i}^{out}\triangleq\{j|j\in\mathcal{V},(i,j)\in\mathcal{E}\}$. The out-degree of $v_{i}$ is $|\mathcal{N}_{i}^{out}|$. The weight adjacency matrix $\mathcal{G}$ is denoted by an $N\times N$ matrix $\mathcal{W}\triangleq[w_{ij}]$ where $w_{ij}$ refers to the weight of the edge from node $i$ to $j$. We assume that graph $\mathcal{G}$ has no self-loops.

We denote the original weight adjacency matrix by $\mathcal{W}^{o}=[w_{ij}^{o}]\in\mathbb{R}^{N\times N}$. Let $\mathcal{N}_{i,o}^{out}$($\mathcal{N}_{i,o}^{in}$) be the set of out-neighbors (in-neighbors) under the original optimal weight pattern $\mathcal{W}^{o}$.

II-B Virus Spreading Model

With the fact that cyber network nodes do not have human-like autoantibody/vaccination which can prevent individual from being infected again, we study the so-called susceptible-infected-susceptible (SIS) models. Consider a population of $N$ agents. Each agent can be either susceptible (S) or infected (I). Infected individuals infect others at rate $\beta_{i}\geq 0$. The intensity of interaction between $v_{i}$ and $v_{j}$ is described by the weight $w_{ij}\in\mathbb{R}$. Denote $\mathbf{w}_{i}=(w_{i1},...,w_{iN})^{\prime}\in\mathbb{R}^{N}$. We assume that the weight is bounded by $\bar{w}_{ij}\in\mathbb{R}$. If $v_{i}\in\mathcal{V}$ is susceptible while $v_{j}\in\mathcal{V}$ is infected, there is possibility that $v_{i}$ will be infected after the interaction. In addition, each infected agent returns to the susceptible state at some rate $\sigma_{i}$. The state of a node $i$ at time $t\geq 0$ is a binary random variable $X_{i}(t)\in\{0,1\}$, with $X_{i}(t)=0$ ($X_{i}(t)=1$), indicating that agent $i$ is susceptible (infected). The state vector of all $N$ agents is denoted by $X(t)=(X_{1}(t),X_{2}(t),...,X_{N}(t))^{\prime}\in\{0,1\}^{N}$. With the adaptive weight $w_{ij}(t)$ from agent $i$ to $j$, the stochastic state transitions of node $v_{i}$ from time $t$ to $t+\Delta t$ can be written as follows:

[TABLE]

The model (1) is computationally challenging under large-scale networks due to the exponentially increasing state space. Hence, we resort to mean-field approximation of the Markov process [17, 21, 22]. Denote $x_{i}(t)\in[0,1]$ as the probability of agent $i$ being infected at time $t$. The mean-field approximation then provides

[TABLE]

for $i=1,2,...,N$. To write this dynamics equation in a more compact form, denote $\mathbf{x}(t)=(x_{1}(t),...,x_{N}(t))^{\prime}$. We have

[TABLE]

where $G(\cdot,\cdot):\mathbb{R}^{N}\times\mathbb{R}^{N\times N}\rightarrow\mathbb{R}^{N}$ which can be written as $G(\mathbf{x}(t),\mathbf{W}(t))=(W(t)B-D)\mathbf{x}(t)-X_{d}(t)W(t)B\mathbf{x}(t)$ where $W(t)={[{w_{ij}(t)}]_{N\times N}}$, $B=diag(\beta_{1},...,\beta_{N})$, $D=diag(\sigma_{1},...,\sigma_{2})$, and $X_{d}(t)=diag(x_{1}(t),...,x_{N}(t))$.

According to the discussion in [17], the $n$-intertwined model (2) gives an upper-bound for the exact probability of infection, $x_{i}(t)$. However, the mean-field approximation consider herein, while it is an approximation, is well constructed because the scale of networks, i.e., N in our model is large and we focus on the cases where $\beta/\sigma$ is above the threshold [22].

The graph and the epidemic spreading process can be viewed as physical constraints. The agents in the network are coupled by these constraints while trying to minimize their own cost. Such behaviors lead to differential games over networks, which will be introduced in the following section.

II-C Differential Game Over Networks

As we mentioned in Section I, the self-interested agents aim to minimize their own costs. One cost arise from malfunction caused by infection. Another cost for agent $i$ is to describe inefficiency or degradation of system performance caused by deviation from the original weight $w_{ij}^{o}$ for all $j\in\mathcal{N}$. We consider the original weight as an optimal weight under which the agent can achieve the most benefit.

For agent $i$, the infection cost function, given by $f_{i}:[0,1]\rightarrow\mathbb{R}^{+}$, is a function of $x_{i}(t)\in[0,1]$. $f_{i}$ is assumed to be monotonically increasing to capture the loss of being infected. A weight cost function for edge from $i$ to $j$ is given by $g_{ij}(w_{ij}(t)-w_{ij}^{o})$ where $g_{ij}:\mathbb{R}\rightarrow\mathbb{R}^{+}$ is convex. The function satisfies $g_{ij}(w)=0$ at and only at $w=0$ for all $i,j\in\mathcal{N}$ because the original weight is optimal to the agent when there is no infection. It is optimal in terms of the tradeoff between price and performance. The marginal cost of deviation from the optimal weight will increase as the distance from the adapted weight to the optimal weight increases. Considering a time duration from [math] to $T\geq 0$, the cost function of agent $i$ during time interval $[0,T]$ is given as follows by

[TABLE]

As each node determines its own weight adaptation policy, it naturally leads to a differential game framework defined as follows. Consider $N$ agents in the network as $N$ players with an index set $\mathcal{N}=\{1,...,N\}$. The duration of the evolution of the game is given by the time interval $[0,T]$. Denote $\mathbf{x}(t)=(x_{1}(t),...,x_{N}(t))^{\prime}$. Let $\mathcal{X}=\{x\in\mathbb{R}^{N}|x_{i}\in[0,1],\forall i\in\mathcal{N}\}$ be the permissible set of the states. For each fixed $t\in[0,T]$, $\mathbf{x}(t)\in\mathcal{X}$. Let $\mathbf{w}_{i}(t)=(w_{i1},...,w_{iN})$ be the controls of player $i$. The admissible control set for player $i$ is $S_{i}=\{0\leq w_{ij}\leq\bar{w}_{ij},\forall j\in\mathcal{N}\}$, i.e., for each fixed $t\in[0,T]$, $\mathbf{w}_{i}\in S_{i}\subset\mathbb{R}^{N}$. A differential equation is given by (2) whose solution describes the state trajectory of the game corresponding to the $N$-tuple of control functions $\{\mathbf{w}_{i}(t),0\leq t\leq T,i\in N\}$ and the given initial state $\mathbf{x}_{0}\triangleq(x_{1}(0),...,x_{N}(0))^{\prime}=(x_{10},...,x_{N0})^{\prime}$. Define a set-valued function $\eta_{i}(\cdot)$ for each $i\in N$ to characterize the information pattern of player $i$. We consider the open-loop pattern in our case where $\eta_{i}(t)=\{\mathbf{x}_{0}\},t\in[0,T]$. We can state our problem as the following differential game problem:

[TABLE]

where $J_{i}(\cdot):\mathbb{R}^{N}\rightarrow\mathbb{R}$ and $x_{i}(0)=x_{i0},\ i=1,2,...,N$. Each player aims to find a control policy $\mu_{i}(t,x_{i0})$ to generate a weight trajectory $w_{i}(t)$. Such control policies are open-loop ones that depend on the initial condition of the individual state.

Remark 1**.**

The game defined by (5) is a differential game over networks where the cost only depends on their own state and controls. Nodes interact with their neighbors. The network topology is captured by $w_{ij},i\in\mathcal{N},j\in\mathcal{N}$. The time-varying property of the network is described by $w_{ij}(t)$ for $t\in[0,T]$.

Remark 2**.**

Information structure determines the state information gained and recalled by players at time $t$. The reasons why we adopt open-loop policies are three-fold. First, the obtained open-loop policy can be implemented as a feedback policy [23] as is shown in Section III. Since the dynamics (3) is determined, the state at any time can be computed and used to determine the control policy. Second, to obtain a strongly time-consistent optimal and individual feedback policies, we have to resort to techniques of dynamic programming. However, a direct application of dynamic programming will not yield an individual feedback policy. Also, computation of the feedback control law derived from Hamilton–Jacobi–Bellman equation requires solving nonlinear PDEs which increases the difficulty of distributed implementation. Third, obtaining open-loop policy resorts to maximum principle which well presents the structure of the optimal solution. This helps us to analyze the inefficiency of the NE and obtain a penalty function to achieve social optimum as is shown in Section IV.

III Analytic Results

The solutions to the $N$-person non-cooperative nonzero-sum differential game (5) played with an open-loop information structure are open-loop Nash equilibria.

Definition 1**.**

The weight adaptation trajectories or say the control trajectories $\{\mathbf{w}^{*}_{i}$, $i\in\mathcal{N}\}$ constitute an open-loop NE solution of the differential game (5) if the inequalities

[TABLE]

hold for all control trajectories $\mathbf{w}_{i}(t)\in S_{i},t\in[0,T]$. We denote $x_{i}^{*}(t),t\in[0,T]$ the associated state trajectory for $i\in\mathcal{N}$.

The definition states that at open-loop NE, no agents have incentive to deviate unilaterally away from the optimal trajectory from time [math] to time $T$.

To obtain the necessary conditions for the open-loop NE, we make two mild assumptions.

Assumption 1**.**

For each $i\in\mathcal{N}$, the infection cost function $f_{i}(\cdot)$ is to be of $C^{1}$ class.

Assumption 2**.**

For each $i,j\in\mathcal{N}$, the weight deviation cost function $g_{ij}(\cdot)$ is to be of $C^{1}$ class.

Each player $i\in\mathcal{N}$ can decide to receive data or packets from any other agent. The following observation narrows down the set of possible solutions of the open-loop NE.

Observation 1**.**

If $\{u^{*}_{ij}(t),i\in\mathcal{N},j\in\mathcal{N}_{i,o}^{out}\}$ is an open-loop NE solution for the following differential game

[TABLE]

with $u_{ij}(t)\in[0,w_{ij}^{o}]$ for $i\in\mathcal{N},j\in\mathcal{N}_{i,o}^{out},t\in[0,T]$, and $\{w_{ij}^{*}(t),i\in\mathcal{N},j\in\mathcal{N}\}$ is an open-loop NE solution for the differential game defined by (5), then we have

[TABLE]

for all player $i$ and for each $t\in[0,T]$.

Proof: See Appendix B-A.

Observation 1 simplifies the searching process for the open-loop NE. Instead of analyzing problem $(\ref{DifGam})$, we can focus on problem (7) which contains a smaller admissible control set. Define $\mathbf{u}_{i}=\{u_{ij},j\in\mathcal{N}_{i,o}^{out}\}$. To be specific, the admissible control set of game problem (7) for player $i$ is $U_{i}=\{0\leq u_{ij}(t)\leq w_{ij}^{o},j\in\mathcal{N}_{i,o}^{out},t\in[0,T]\}$. From Theorem 5.1 of [23] and Lemma 1, the differential equation in (7) admits a unique solution if the weight adaptation control is continuous in $t$.

Next, we discuss the derivation of candidate NE solutions for differential game (7) when the information structure of the game is open-loop pattern. Utilizing techniques in optimal control theory, we arrive at the following result.

Theorem 1**.**

For the $N$-person differential game (7), we have assumptions 1 and 2. Then, if $\{\mathbf{u}^{*}_{i}(t),i\in\mathcal{N}\}$ is an open-loop NE solution, and $\{\mathbf{x}^{*}(t),0\leq t\leq T\}$ is the corresponding state trajectory, there exist $N$ costate functions $\mathbf{p}_{i}(\cdot):[0,T]\rightarrow\mathbb{R}^{N},i\in\mathcal{N}$, whose $j$-th component is denoted by ${p}_{ij}(\cdot)$, such that the following relations are satisfied:

[TABLE]

[TABLE]

[TABLE]

where

[TABLE]

and $\Gamma_{i}$ is a matrix given by

[TABLE]

$\gamma_{i}$* is a vector whose $i$-th component is $-df_{i}/dx_{i}$ and other components are zero, for $i\in\mathcal{N}$.*

Proof: See Appendix B-B.

Note that $\Gamma_{i}$ turns out to be the same for different $i$. In later discussion, we shall omit the idex $i$. Now, the dynamics of the costate function can be given as $\dot{\mathbf{p}}_{i}(t)=\Gamma(t)\mathbf{p}_{i}(t)+\gamma_{i}(t)$ for $i\in\mathcal{N}$ which sheds some light on the design for achieving social welfare in the following section. $\Gamma(t)$ is a $L$-matrix [24] for every $t\in[0,T]$ where the diagonal entries of $\Gamma(t)$ are positive and all off-diagonal entries are non-positive. Therefore $\Gamma(t)$ is structurally in line with the graph Laplacian whose diagonal entries are the out-degrees of the $N$ agents [25]. That is for every zero or negative entry of the matrix $\Gamma(t)$, the corresponding entry of the graph Laplacian is zero or negative respectively and vice versa. If the original graph is a directed acyclic graph, $\Gamma(t)$ is a lower triangular matrix given the index of a proper permutation. Other than the topology information, $\Gamma(t)$ also contains the infection information. Note that even though we write the dynamics of the costates in an affine form, it is actually not affine which is because $\Gamma(t)$ depends on $\mathbf{x}^{*}(t)$ and $u^{*}_{ij},i\in\mathcal{N},j\in\mathcal{N}$ as we can see from (13) and $u^{*}_{ij}$ is dependent on $p_{ii}$ as we will show next in Theorem 2.

Theorem 2**.**

Define $\phi_{ij}(t):=p_{ii}(t)(1-x_{i}^{*}(t))\beta_{j}x_{j}^{*}(t)$ where $p_{ii}(\cdot)$ is the $i$th component of the costate function $\mathbf{p}_{i}(\cdot)$. The basic structure of the NE-based optimal weight control, i.e., the solution to (10), can be written as:

[TABLE]

for $i\in\mathcal{N},j\in\mathcal{N}_{i,o}^{out}$.

Proof: See Appendix B-C.

Condition (10) in Theorem 1 can thus be replaced by $(\ref{ConRule})$. Theorem 1 together with (14) provides a weight adaptation scheme where each agent adapts its weight to minimize the possibility of being infected and the loss of efficiency/interest. The weight of the edge from player $i$ to player $j$, controlled by player $i$, is based on the costate component $p_{ii}(t)$, player $i$’s own infection $x_{i}(t)$ and its out-neighbors infection. Apparently, the higher the infection level of agent $j$ is, the lower the weight of edge $(i,j)$ should be. As is shown in (11) and (13), $p_{ii}(t)$ is highly coupled and it contains information about the effect of the whole network.

Remark 3**.**

Based on the structure of the optimal control (14), the dynamics of costates (11) as well as Lemma 1, we can infer that the NE-based optimal control trajectory $u^{*}_{ij}(t)$ is continuous for every $i\in\mathcal{N},j\in\mathcal{N}$ which means there is no switching in the optimal weight adaptation.

Remark 4**.**

From (14), we know the weight between agents $i$ and $j$ may be adapted to zero at certain time as one can see that $u^{*}_{ij}(t)=0$ if $-\phi_{ij}(t)\leq g^{\prime}_{ij}(-\omega^{o}_{ij})$. That means the connection between agent $i$ and $j$ may be disconnected temporarily which will be restored according to Theorem 3. For agent $i$, if all its out-links have weight zero, i.e., $-\phi_{ij}(t)\leq g^{\prime}_{ij}(-\omega^{o}_{ij})$ for all $j\in\mathcal{N}_{i,o}^{out}$, we can view this agent as being quarantined from infection. We say being quarantined from infection because there might still be in-links connecting to agent $i$ which means here, the concept of being quarantined is different from the concept in undirected graph. Besides, the weight adaptation scheme we proposed is different from quarantining in a sense that the weight adaptation scheme does not need to completely disconnect nodes from all other others but rather adjust weights to connect more loosely with particular nodes with a higher likelihood of infection.

Corollary 1**.**

If $g_{ij}(\cdot)$ is concave, i.e., the marginal cost of deviation increases as the adapted weight becomes more far away from the optimal weight, the optimal control policy can be given as follows

[TABLE]

for $i\in\mathcal{N},j\in\mathcal{N}_{i,o}^{out}$, where $\phi_{ij}(t)$ is defined in Theorem 2.

We can see that if $g_{ij}(\cdot)$ is concave, the optimal control policy switches between [math] and $w_{ij}^{o}$. In this paper, we focus our study on the case when $g_{ij}(\cdot)$ is convex.

Before stepping into the numerical computation of the open-loop NE candidates, we go into further analysis and obtain other structural results that would be beneficial for more insightful understanding of the weight adaptation mechanism.

Theorem 3**.**

The costate function and the open-loop control trajectories have the following properties:

• (i)

Along the open-loop NE trajectory, $p_{ij}(t)\geq 0$ holds for all $i,j\in\mathcal{N},j\neq i$ and all $t\in[0,T]$. Furthermore, $p_{ii}(t)$ stays positive for all $i\in\mathcal{N}$ and all $t\in[0,T)$.

• (ii)

The open-loop NE control trajectory $u_{ij}^{*}(t)$, for $i\in\mathcal{N},j\in\mathcal{N}_{i,o}^{out}$, satisfies $u_{ij}^{*}(t)=w_{ij}^{o}$ at and only at $t=T$ and for $t<T$, $u_{ij}^{*}(t)<w_{ij}^{o}$.

• (iii)

If $|\mathcal{N}_{i,o}^{in}|=0$, i.e., the in-degree of player $i$ is zero in the original graph, under linear infection cost function $f_{i}(x_{i}(t))=\alpha_{i}x_{i}(t)$, the component $p_{ii}(t)$ is bounded above by $\alpha_{i}/\sigma_{i}$. That is, $p_{ii}(t)\leq\alpha_{i}/\sigma_{i}$ for $t\in[0,T]$.

• (iv)

If $|\mathcal{N}_{i,o}^{out}|=0$, i.e., the out-degree of player $i$ is zero in the original graph, under linear infection cost function where $f_{i}(x_{i}(t))=\alpha_{i}x_{i}(t)$, the costate component $p_{ii}(t)$ is strictly monotonically decreasing over $t$.

Proof: See Appendix B-D.

Theorem 3 indicates that during the time interval $[0,T)$, the agents, with an incentive to lower their own costs, adapt their weight accordingly to impede the spreading of virus. After the prescribed alert duration $[0,T]$, a recovery of topology is always on the way to meet the minimum cost. Also, from theorem 3 (iii), we know for agent $i$ who has no in-neighbors, its out-link $u_{ij}^{*}(t)$ will never be [math] if $\alpha_{i}\beta_{j}/\sigma_{i}\leq g_{ij}^{\prime}(-w_{ij}^{o})$. This can be readily shown by $\phi_{ij}=p_{ii}(1-x_{i}^{*}(t))\beta_{j}x_{j}^{*}(t)\leq(\alpha_{i}/\sigma_{i})(1-x_{i}^{*})\beta_{j}x_{j}^{*}(t)<\alpha_{i}\beta_{j}/\sigma_{i}\leq g_{ij}^{\prime}(-w_{ij}^{o})$.

IV Inefficiency of Nash Equilibrium

It is well known that the non-cooperative NE in nonzero-sum games is generally inefficient [26]. There is need to develop a mechanism to attain a higher social welfare or lower aggregate costs through cooperation behavior [27]. The notion of the price of anarchy has been introduced in [28] to quantify the inefficiency. In the network, the social cost is the aggregate costs of all players. Let $\mathbf{u}=\{\mathbf{u}_{1},...,\mathbf{u}_{N}\}$ where $\mathbf{u}_{i}(t)\in\mathbb{R}^{|\mathcal{N}_{i,o}^{out}|}$ be the weight control variable for the whole network with admissible set $U_{o}=\{\mathbf{u}:u_{ij}(t)\in[0,\omega_{ij}^{o}],\forall i\in\mathcal{N},j\in\mathcal{N}_{i,o}^{out},t\in[0,T]\}$. Denote by $\mathbf{u}^{o}=\{\mathbf{u}_{1}^{o},...,\mathbf{u}_{N}^{o}\}$ the social optimal solution. The social optimum can be attained by solving the optimal control problem:

[TABLE]

where $J_{o}:\mathbb{R}^{|\mathcal{N}_{1,o}^{out}|}\times\cdots\times\mathbb{R}^{|\mathcal{N}_{N,o}^{out}|}\rightarrow\mathbb{R}$. An application of maximum principle gives the following: the optimal control $\mathbf{u}^{o}(t)$ and corresponding trajectory $\mathbf{x}^{o}(t)$ must satisfy the following so-called canonical equations:

[TABLE]

for all $i\in\mathcal{N}$, where $\Gamma(t)$ is the same with the one given in (13) for the dynamics of the costate in the differential game problem and $\gamma(t)=[-f^{\prime}_{1}(x_{1}(t)),-f^{\prime}_{2}(x_{2}(t)),...,-f^{\prime}_{N}(x_{N})]^{\prime}$, the Hamiltonian of the optimal control problem is defined as

[TABLE]

and $\lambda(\cdot):[0,T]\rightarrow\mathbb{R}^{N}$ is the costate function, $\lambda_{i}$ is its $i$th component. The Hamiltonian of the optimal control problem (20) is different from the individual Hamiltonian defined in (12). But they are related. The Hamiltonian of the optimal control problem includes the cost of all agents over the network instead of just individual’s cost. Also, the costate $\lambda(t)$ corresponds to the state of all agents $\mathbf{x}(t)$. The counterpart of $\lambda_{j}(t),j\in\mathcal{N}$ in the individual Hamiltonian defined for the game problem is $p_{ij}(t),j\in\mathcal{N}$. Due to the similar structure of the Hamiltonian of the optimal control problem and the individual Hamiltonians for the game problem, after applying maximum principle, we obtain (17-19) that are in the same structure with (9-11).

An optimal point can in principle be computed centrally by network operator to achieve social optimum. However, this will require the network operator to be omniscient and also not all the agents have incentives to adapt their connection weights based on the rule designed to minimize the aggregate costs. Also, for large-scale network/system, centralized solution gives rise to computational problems and implementability issues. So, centralized optimal control solution is impractical. To achieve social optimum in a distributed way, a mechanism needs to be designed on behalf of the network operator. The strategy for the network operator is to set penalties $c_{i}:=(c_{i}(t),t\in[0,T])$ so that the cost for player $i$ at time $t$ is $\hat{l}_{i}(t,\mathbf{x},\mathbf{u})\triangleq f_{i}(x_{i}(t))+\sum_{j\in\mathcal{N}_{i,o}^{out}}g_{ij}(u_{ij}(t)-w^{o}_{ij})+c_{i}(t)$. To set a proper penalty for each player, we need to utilize the theory of potential differential games [29] which is an extension of the potential game concept for static games [30]. The following is the definition of potential differential games.

Definition 2**.**

A differential game with cost $\hat{J}_{i}=\int_{0}^{T}\hat{l}_{i}(t,\mathbf{x},\mathbf{u})dt$ and dynamics defined by (2) is an potential differential game if there exists a function $\pi:\mathcal{X}\times U_{1}\times\cdots\times U_{N}\times[0,T]\rightarrow\mathbb{R}$ that satisfies the following condition for every player $i\in\mathcal{N}$

[TABLE]

for all $\mathbf{u}_{i},\mathbf{v}_{i}\in U_{i}$, where $\mathbf{x},\hat{\mathbf{x}}\in\mathcal{X}$ are the corresponding states under controls $\{\mathbf{u}_{i},\mathbf{u}_{-i}\}$ and $\{\mathbf{v}_{i},\mathbf{u}_{-i}\}$ respectively. Here, $\mathbf{u}_{-i}$ denote the collection of controls of all players except player $i$, i.e., $\mathbf{u}_{-i}=\{\mathbf{u}_{1},...,\mathbf{u}_{i-1},\mathbf{u}_{i+1},...,\mathbf{u}_{N}\}$.

If we can find $c_{i}(t)$ for every $i\in\mathcal{N}$ such that relation (21) holds for $\pi=\sum_{i=1}^{N}f_{i}(x_{i}(t))-\sum_{i=1}^{N}\sum_{j\in\mathcal{N}_{i,o}^{out}}g_{ij}(u_{ij}(t)-w^{o}_{ij})$, then the differential game is a potential game whose open-loop NE solution is equivalent to the open-loop solution of optimal control problem defined by (16). The following theorem gives important insights about choosing proper penalties $c_{i}(t)$.

Theorem 4**.**

Consider a differential game with penalties where the cost of player $i$ is given by

[TABLE]

which is obtained by introducing a penalty term introduced to $J_{i}$ in (7), and the constraint dynamics is in accordance with the differential game defined by (7). Let $c_{i}(t)=\sum_{j\neq i}f_{j}(x_{j}(t))$. Then, the differential game with (22) is a potential differential game corresponding to the optimal control problem defined by (16), and if $\{\mathbf{u}^{*}_{i}(t),i\in\mathcal{N}\}$ is an open-loop NE solution for new differential game with cost (22), and $\{\mathbf{x}^{*}(t),0\leq t\leq T\}$ is the corresponding state trajectory, the relations (17) (18) and (19) hold for $\mathbf{u}^{*}$ and $\mathbf{x}^{*}$ with $\mathbf{u}^{o}$ replaced by $\{\mathbf{u}^{*}_{i}(t),i\in\mathcal{N}\}$ and $\mathbf{x}^{o}$ replaced by $\mathbf{x}^{*}$.

Proof: See Appendix B-E.

Theorem 4 indicates that with a proper choice of the penalties $c_{i}(t),i\in\mathcal{N}$, the necessary conditions for the open-loop NE solution of the differential game is aligned with the necessary conditions for the optimal solution of the social cost optimal control problem. The counterpart of condition (11) for the penalty-based differential game can be written as

[TABLE]

where $j$th component of $\hat{\gamma}_{i}(t)$ is $\partial(f_{i}+c_{i})/\partial x_{j}$. In implementation, the system operator sends the penalty function to each agent. If the open-loop NE and the social optimal solution uniquely exist, the open-loop NE achieves the social optimum.

Definition 3**.**

A directed graph is strongly connected if it contains a directed path from $j$ to $i$ for every pair of vertices $\{(i,j):i\in\mathcal{N},j\in\mathcal{N},i\neq j\}$.

In a directed graph, a directed path is a sequence of edges which join a sequence of vertices, but with the restriction that the edges all be directed in the same direction. What we have developed so far in this section is for cases where the original weighted network is strongly connected. If the original network is not strongly connected, we have the following.

Definition 4**.**

Given a graph, if there exists a directed path from vertex $j$ to vertex $i$, we say $i$ is reachable from $j$. Denote by $\mathcal{R}_{i}\subseteq\mathcal{N}$ the set of vertices that $i$ can be reachable from.

In graph theory, a single vertex is defined to connect to itself by a trivial path. We have $i\in\mathcal{R}_{i}$. If the graph is strongly connected, for every $i\in\mathcal{N}$, $\mathcal{R}_{i}=\mathcal{N}$. Otherwise, for some $i\in\mathcal{N}$, we have $\mathcal{R}_{i}\subset\mathcal{N}$. Denote $\mathcal{R}_{i,o}$ the counterpart of $\mathcal{R}_{i}$ for the original graph defined by $(\mathcal{V},\mathcal{E},\mathcal{W}^{o})$.

Corollary 2**.**

Consider the differential game with cost functions defined in (22) and dynamics given by (7) as in Theorem 4. Let $c_{i}(t)=\sum_{j\in\mathcal{R}_{i,o}\backslash\{i\}}f_{j}(x_{j})$. Then, if $\{\mathbf{u}^{*}_{i}(t),i\in\mathcal{N}\}$ is an open-loop NE solution for the new differential game, and $\{\mathbf{x}^{*}(t),0\leq t\leq T\}$ is the corresponding state trajectory, the relations (17), (18), and (19) hold for $\mathbf{u}^{*}$ and $\mathbf{x}^{*}$ with $\mathbf{u}^{o}$ replaced by $\{\mathbf{u}^{*}_{i}(t),i\in\mathcal{N}\}$ and $\mathbf{x}^{o}$ replaced by $\mathbf{x}^{*}$.

Proof.

The proof of Corollary 2 simply follows from the proof for Theorem 4. ∎

To illustrate Corollary 2, we present an example in Appendix C

V Algorithms and Case Studies

In this section, we provide the set-up information for the case studies. Besides, based on the equilibrium analysis and the optimal control analysis, an algorithm is proposed to compute the optimal weight adaptation trajectory for the system operator and the agents.

V-A Preliminaries

In the simulation, the infection cost function $f_{i}(\cdot)$ is given to be linear in $x_{i}(t)$, i.e., $f_{i}(x_{i}(t))=\alpha_{i}x_{i}(t)$. Here, we set $\alpha_{i}=\alpha,\forall i\in\mathcal{N}$. The weight adaptation cost is taken to be quadratic where $g_{ij}(u_{ij}-w_{ij}^{o})=(1/2)d_{ij}(u_{ij}-w_{ij}^{o})^{2}$ for all $i\in\mathcal{N},j\in\mathcal{N}_{i,o}^{out}$. Unless otherwise stated, let $\alpha_{i}=1,d_{ij}=d=0.2$ for all $i\in\mathcal{N},j\in\mathcal{N}_{i,o}^{out}$. Note that under this setting, assumptions 1 and 2 hold and $g_{ij}(\cdot)$ is even and convex.

The original network in the simulation is a bi-directional scale-free network with $150$ agents generated based on the Barabási-Albert model [31, 32]. We select this model since many kinds of computer networks, including the internet and the web graph of the World Wide Web, have scale-free properties. We generate the network by following the growth and preferential attachment properties given in section VII of [31]. For simplicity, the original weight is set to be $w^{o}_{ij}=1$ for all edge $(i,j)$. Let $\langle k^{in}\rangle$ $\langle k^{out}\rangle$ be the average in-degree (out-degree) of the network we generated. We have $\langle k^{in}\rangle=\langle k^{out}\rangle=7.545$.

For simplicity, we take same infection rates and curing rates for all players. Unless otherwise stated, let $\beta_{i}=\beta=0.04$ and $\sigma_{i}=\sigma=0.1$. From the result in [34] and the fact that the largest real part of the eigenvalues of matrix $(\mathcal{W}^{o}B-D)$ is $0.5368$, we say that the virus epidemic will outbreak in the original network. The initial infection level is also set to be the same for all players, $x_{i0}=0.16$ for all $i\in\mathcal{N}$. Table I is a summary of the setups.

V-B Computational Algorithm

Note that we aim to propose an implementable distributed virus resistance algorithm (DVR algorithm). Based on the algorithm proposed in [33] for computation of open-loop NE for nonzero-sum differential games, we present, in algorithm 1, the DVR algorithm to compute the candidate open-loop NE solutions for the differential game described by (7) and the penalty-based differential game defined in Theorem 4. The solution of the penalty-based differential game is inline with the solution of the optimal control problem defined in (16).

Initially, the input data includes initial infection data: $x_{i0}$ for all $i\in\mathcal{N}$; infection rate $\beta_{i}$, recovery rate $\sigma_{i}$ for all $i\in\mathcal{N}$; the original topology $\mathcal{W}^{o}$; the cost functions $f_{i}(\cdot),g_{ij}(\cdot)$ for all $i\in\mathcal{N},j\in\mathcal{N}_{i,o}^{out}$ and a stopping value $\epsilon$ to stop the algorithm. In the first step, each player arbitrarily selects a continuous control trajectory within the admissible control set $U_{i}$ for every out-link it has: $u_{ij}(t)$ for each $i\in\mathcal{N}$, for all $j\in\mathcal{N}_{i,o}^{out}$, and reports the weight adaptation scheme $\mathbf{u}_{i}(t)$ to the network operator. In step $2$, each player utilizes the initial infection data $x_{i0},i\in\mathcal{N}$ and the control policy $\mathbf{u}_{i},i\in\mathcal{N}$, solve (9) forward in time to obtain $x_{i},i\in\mathcal{N}$ and report it to the network operator. If the system aims to achieve the social optimal control problem, then the algorithm goes into step $3$. Otherwise, the algorithm steps into step $3^{\prime}$. In step $3$, the system operator utilizes the reported $\mathbf{u}_{i}$ and $x_{i},i\in\mathcal{N}$, the infection damage cost $f_{i}(\cdot),i\in\mathcal{N}$ to compute $\mathbf{p}(t)$ backward based on (23) and sends $\mathbf{p}_{i}(t)$ back to the corresponding player $i$. In step $3^{\prime}$, the system operator utilizes the reported $\mathbf{u}_{i}$ and $x_{i},i\in\mathcal{N}$, the infection damage cost $f_{i}(\cdot),i\in\mathcal{N}$, computes $\mathbf{p}_{i}(t),i\in\mathcal{N}$ backward based on (11) and sends $\mathbf{p}_{i}(t)$ back to the corresponding player $i$. In the next step, each player updates its control based on (14) which only requires its out-neighbors infection information and reports the updated control policy to the network operator. Denote by $\hat{\mathbf{u}}_{i}(t)$ the updated control policy. If $\|{\hat{\mathbf{u}}-\mathbf{u}}\|_{\infty}\geq\epsilon$, the algorithm moves back to step $2$. Otherwise, the latest updated policy $\mathbf{\hat{u}}_{i}$ is the optimal control policy for agent $i$.

V-C Numerical Results

In this subsection, we present the numerical results. First, we show the dynamics of the costate function for all players. Then, we show the evolution of the weight adaptation, the infection and the costate of selected agents to see individuals’ behaviors. Second, we give the comparisons between the optimal control based-weight adaptation scheme (this scheme is equivalent to the penalized differential game based-weight adaptation scheme) and the differential game-based weight adaptation scheme. The optimal control based adaptation scheme is from solving optimal control problem (16). The two schemes together with the case of weight adaptation. are compared in terms of the total cost and the infection level of the whole network.

From $(\ref{ConRule})$ and $\phi_{ij}(t):=p_{ii}(t)(1-x_{i}^{*}(t))\beta_{j}x_{j}^{*}(t)$, we know that the weight adaptation of player $i$ is based on its own infection, its out-neighbors, and the costate component $p_{ii}$. The infection of player $i$ and its neighbors are just local information. From (11) and (13), we can see the effect of the whole network’s situation is conveyed by costate component $p_{ii}$ to the weight adaptation strategy of player $i$. Thus, we investigate the dynamics of $p_{ii}$ in Fig. 4, where the costate component $p_{ii}$’s dynamics for all agents are plotted. As we can see, $p_{ii}(t)$ is positive for all $i\in\mathcal{N}$ during the whole time interval which corroborates Theorem 3. For most of the players, the value of $p_{ii}$ is high at the very beginning and then decreases to [math]. One interpretation is that players are more sensitive at the beginning to their out-neighbors infection and tend to cut their weights more heavily.

To see individual behaviors and states, we rank the agents based on their out-degrees. Agent $1$ has the largest out-degree. From the first plot of Fig. 5, agent $1$ is more likely to be infected due to its large degree. We can see that all weights equal $1$ at and only at $t=T=20$, which corroborates Theorem 3. The weight $u_{150,1}(t)$ is reduced to [math] for some time. This phenomenon occurs because the costate and its out-neighbors’ infection levels are high during that time period. The third plot shows that agents with higher out-degrees reduce less weight. Usually, one suppose to cut more weights on highly connected nodes to slow the infection propagation. However, the obtained weight adaptation scheme in this paper is a result of considering both the infection and the loss of efficiency of the network agents. There is a trade-off between maintaining the network’s performance and lowering the infection. So, the agents with higher out-degrees may cut less weight to maintain the performance/function.

To show that each agent has heterogeneous weight adaptation to different neighbors, we present Fig. 6. We can know from Fig. 6 that agent $1$ adapts weights with his/her out-neighbors accordingly based on the evolution of the infection levels of his/her out-neighbors. As we can see, agent $1$ cuts more weight on neighbors with higher infection levels. For example, the infection level of agent $2$ is higher than agent $150$ all the time. Thus, weight $u_{1,2}$ is lower than $u_{1,150}$. Also, agent $83$ reduces its weight on agent $1$ to zero due to the latter’s high infection level while its weight on agent $93$ remains above $0.5$.

Here, we compare the NE-based weight adaptation scheme, the optimal control-based weight adaptation scheme, and no weight adaptation scheme. In Fig. 7, we plot the total cost $J_{o}$ under the three schemes for different $\alpha$. We observe that no adaptation scheme cause the most total cost. For different values of $\alpha$, the NE-based scheme always incurs a higher cost than the optimal control-based scheme, which indicates the inefficiency of the NE solution. From the plot, we see that a higher $\alpha$ causes more inefficiency.

Fig. 8 is presented to show the virus-resistance of the proposed schemes. The black line shows the infection level for the case with no adaptation scheme, the blue line shows the case with the game-based scheme, and the green line shows the case with the optimal control-based scheme. Even though the game-based scheme is inefficient in terms of minimizing the total cost, it outperforms the optimal control-based scheme since the infection level under the game-based scheme is always lower than the infection level under the optimal control based scheme. No matter in what case, the scheme we have proposed has proven to be virus-resistant and generated a lower total cost than the scheme without adaptation did.

VI Conclusion and Future Work

In this paper, we have established a differential game framework to develop decentralized virus-resistant mechanisms over complex networks. We have shown that weight adaptation policies allow nodes to change weights to mitigate their infection. The differential game approach has captured the strategic and dynamic behaviors of a large number of self-interested agents over time-varying networks. Each player adapts its weight based on its own infection and its out-neighbors infection. It has been observed that the higher levels of its out-neighbors’ infection lead to lower weights. The effect of non-local behaviors on the adaptation strategy has been encoded in the costate function. We have discussed the inefficiency of the open-loop Nash equilibrium and have proposed a penalty-based mechanism to achieve efficiency by imposing local costs induced by reachable nodes. The differential game framework has enabled the design and implementation of a distributed algorithm over large-scale networks to control the macroscopic behaviors of the virus spreading over networks. Numerical examples have been used to illustrate the virus-resistance of the proposed scheme and the inefficiency of the Nash equilibrium. The differential game approach achieves a better performance than its centralized counterpart in terms of the mitigation of virus spreading. One future direction for this work would be to study the steady behavior of long-term virus-resistance scheme where the duration of virus spreading is sufficiently long.

Appendix A Lemmas

Lemma 1**.**

The dynamics equation $G(\mathbf{x}(t),\mathbf{W}(t))$ is uniformly Lipschitz in $\mathbf{x}$ and $\mathbf{w}_{i}$ for each $i\in\mathcal{N}$.

Proof.

From (3), we have

[TABLE]

where ${\beta_{M}}\coloneqq{\max_{i}}{\beta_{i}}$ and ${\delta_{M}}$ $\coloneqq$ ${\max_{i}}{\delta_{i}}$, and $I_{N}\coloneqq\textrm{diag}(1,1,\cdots,1)$. Since $w_{ij}$ is bounded by $\bar{w}_{ij}$, $(1+{\left\|{W(t)}\right\|_{\infty}})$ is bounded. So, $G$ is uniformly Lipschitz in $\mathbf{x}$. The proof for $G$ is uniformly Lipschitz in $\mathbf{w}_{i}$ for all $i\in\mathcal{N}$ can be obtained by following the similar steps. ∎

Lemma 2**.**

Let $x_{i}$, $i\in\mathcal{N}$ be the corresponding solution to the ODEs in (2). For all $i\in\mathcal{N}$, given $0<x_{i}(0)\leq 1$, $x_{i}(t)\in(0,1)$ holds for all $t\in(0,\infty)$.

Proof.

The proof follows Lemma 1 of [17]. It is clear that $x_{i}(t)$ is a continuous function of time. When $x_{i}(0)=1$, then from (2), we have $\dot{x}_{i}(0)<0$, which means once $x_{i}(t)$ reaches $1$, it cannot stay there. On the other hand, when $x_{i}(0)\in(0,1)$, the solution $x_{i}(t)$ would always lie in $(0,1)$. Otherwise, suppose that there exists $t_{1}$ such that $x_{i}(t_{1})=0$ or $x_{i}(t_{1})=1$. In the first case, note that $\dot{x}_{i}(t)\geq-\sigma_{i}x_{i}(t)$ holds for the time interval $(0,t_{1}]$, which gives $x_{i}(t_{1})\geq x_{i}(0)e^{-\sigma_{i}t_{1}}>0$. It yields a contradiction. In the second case where $x_{i}(t_{1})=1$, we have $x_{i}(t)<1$ over time interval $[0,t_{1})$. So, we have $\dot{x}_{i}(t_{1}^{-})\geq 0$ which contradicts the fact obtained from (2) that $\dot{x}_{i}(t_{1})<0$. ∎

Appendix B proof

B-A Proof of Observation 1

Proof.

Given the original weight pattern $\mathcal{W}^{o}$, suppose that $w_{ij}^{o}=0$, i.e., $j\notin\mathcal{N}_{i,o}^{out}$ and $w^{*}_{ij}(t)>0$. Obviously, player $i$ can lower his own cost by deviating $w_{ij}(t)$ from $w^{*}_{ij}(t)$ to [math], which contradicts the fact that $w^{*}_{ij}(t)$ is the open-loop NE. A similar statement can be made for the case when $w^{*}_{ij}>w_{ij}^{o}$. ∎

B-B Proof of Theorem 1

Proof.

Based on Theorem 6.11 in [23], conditions (9) and (10) are directly derived. To obtain (11), we have

[TABLE]

Reformulating (25), we obtain condition (11). As the terminal cost is unspecified and the final state is free, we have the transversality condition $\mathbf{p}_{i}(T)=0$. ∎

B-C Proof of Theorem 2

Proof.

By Assumption 2 and the fact that $g_{ij},i,j\in\mathcal{N}$ is convex, we know that the Hamiltonian $H_{i}$ is differentiable and convex on $u_{ij}$ for every $j$. Also, the admissible control set is convex. Thus, the solution of (10) can be obtained by letting $\partial H_{i}(u_{ij})/\partial u_{ij}=0$, i.e.,

[TABLE]

Suppose ${\partial H_{i}(\tilde{u}_{ij}(t))}/{\partial u_{ij}(t)}=0$. If $\tilde{u}_{ij}(t)\in[0,\omega_{ij}^{o}]$ which happens if and only if $g^{\prime}_{ij}(-w_{ij}^{o})<-\phi_{ij}(t)<g^{\prime}_{ij}(0)$, then according to (10), $u^{*}_{ij}(t)=\tilde{u}_{ij}(t)=(g^{\prime}_{ij})^{-1}(-\phi_{ij}(t))$. Otherwise, if $\tilde{u}_{ij}(t)<0$, $u^{*}_{ij}(t)=0$ while if $\tilde{u}_{ij}(t)>\omega_{ij}^{o}$, $u^{*}_{ij}(t)=\omega_{ij}^{o}$. Thus, we have the optimal control rule in the form of (14). ∎

B-D *Proof of Theorem 3 *

Proof.

For proof of (i), note the fact that if $h(x)$ is a continuous and piece-wise differential function over $[a,b]$ such that $h(a)=h(b)$ while $h(x)\neq h(a)$ for all $x$ in $(a,b)$, $(dh/dx)(a^{+})$ and $(dh/dx)(b^{-})$ cannot be negative simultaneously. From (11), we have $\mathbf{p}_{i}(T)=0$, which gives $\dot{p}_{ii}(T^{-})=\lim_{t\rightarrow T^{-}}\dot{p}_{ii}(t)=-f^{\prime}_{i}(x_{i}(T))<0$. Hence, there exists $\epsilon_{i}>0$ such that $p_{ii}>0$ and $p_{ij}\geq 0$ for $j\in\mathcal{N}/\{i\}$ over $[T-\epsilon_{i},T]$. Suppose that one of the costate component $p_{ij}$ violates the inequality first at $t_{a}<T$, i.e., we have $p_{ij}(t)\geq 0$ for $t\geq t_{a}$, we obtain $\dot{p}_{ij}(t_{a}^{+})\leq 0$ which is not feasible. If it is $p_{ii}$ that first violates the inequality at time $t_{a}$, we have $\dot{p}_{ii}(t_{a}^{+})=-f^{\prime}_{i}(x^{*}_{i}(t))-\sum_{j\in\mathcal{N}_{i,o}^{in}}p_{ij}(t)(1-x^{*}_{j}(t))u^{*}_{ji}(t)\beta_{i}<0$ which is in contradiction with the fact.

To prove (ii), note that at time $T$, we have $\mathbf{p}_{i}(T)=0$. Combined with the fact that $g_{ij}(\cdot)$ is convex and continuously differentiable, $g_{ij}(0)=0$ and $p_{ii}(t)>0$, by expression (14), we have that $u^{*}_{ij}(t)=w^{o}_{ij}$ holds only at $t=T$.

For proof of (iii), from Observation 1, we know that if $w^{o}_{ji}=0$, the optimal weight adaptation $u^{*}_{ji}(t)=0$ for every $t$. Here, $|\mathcal{N}^{in}_{i,o}|=0$ indicates $w^{o}_{ji}=0$ for all $j\in\mathcal{N}$. Based on (11) and (13), the dynamics of the costate component $p_{ii}(t)$ can be written as

[TABLE]

Note that the first term in the bracket is non-negative and $p_{ii}(T)=0$. Moving backward from $T$, it’s obvious that $p_{ii}(t)$ is bounded above by $\alpha_{i}/\sigma_{i}$.

Under conditions stated in (iv), the dynamics of the costate component $p_{ii}(t)$ can be written as

[TABLE]

We have proved in (i) that $p_{ij}(t)\geq 0$ holds for all $i,j\in\mathcal{N},i\neq j$ and for $t\in[0,T]$ which indicates $\dot{p}_{ii}<0$. Thus, $p_{ii}(t)$ is strictly monotonically decreasing over $t$. ∎

B-E Proof of Theorem 4

Proof.

Assume that $\mathbf{u}_{o}=(\mathbf{u}_{1}^{o},...,\mathbf{u}_{n}^{o})$ is an open-loop optimal control of the centralized control problem (16), and $\mathbf{x}_{o}=(\mathbf{x}_{1}^{o},...,\mathbf{x}_{n}^{o})$ is the state path under the optimal control. Fix an arbitrary $i\in\mathcal{N}$, and let $\mathbf{u}_{i}\neq\mathbf{u}_{i}^{o}$ be an open-loop strategy for player $i$. Let $\mathbf{x}=(\mathbf{x_{1}},...,\mathbf{x}_{n})$ be the new state trajectory given by (16) corresponding to $(\mathbf{u}_{i},\mathbf{u}_{-i}^{o})$. As $\mathbf{u}^{o}$ and $\mathbf{x}^{o}$ are optimal for the optimal control problem (16), then

[TABLE]

Adding to both sides of this this inequality the constant

[TABLE]

we obtain that $\hat{J}_{i}(\mathbf{u}_{i},\mathbf{u}_{-i}^{o})\geq\hat{J}_{i}(\mathbf{u}^{o})$ for all $\mathbf{u}_{i}\in U_{i}$. According to the definition of open-loop NE for differential games in (6), we know $\mathbf{u}^{o}$ is also an open-loop NE for the differential game with penalties.

To show the optimal control problem (16) shares the same necessary conditions with the new differential game, we again utilize the maximum principle. The Hamiltonian of player $i$ for the new differential game is $\hat{H}_{i}=H_{i}+c_{i}(t)$. We can find that relations (9) (10) and (11) under the Hamiltonian $\hat{H}_{i}$ are aligned with relations (17) (18) and (19) where $\lambda(t)=\mathbf{p}_{i}(t)$ at each $t$ for all $i\in\mathcal{N}$. ∎

Appendix C Example

To illustrate Corollary 2, we consider a directed network in Fig. 3. Here, $\mathcal{R}_{1}=\{1\}$, $\mathcal{R}_{3}=\{1,2,3,4\}$, $\mathcal{R}_{5}=\{1,4,5\}$. The $\Gamma(t)$ associated with this network can be rewritten as an upper triangular block matrix. The upper triangular matrix is denoted by $\Gamma_{\mathcal{R}_{i}}$ where the first $|\mathcal{R}_{i}|$ rows and columns of this matrix represent the vertices in $\mathcal{R}_{i}$ in an ascending order. The last $N-|\mathcal{R}_{i}|$ rows and columns represent the rest of the vertices in $\mathcal{N}\backslash\mathcal{R}_{i}$ in an ascending order. For example, the permutation for agent $5$ is $\{1,4,5,2,3\}$. Thus, the dynamics of $\mathbf{p}_{5}$ under the differential game given in Corollary 2 can be written as

[TABLE]

where

[TABLE]

Thus, if we let $c_{i}(t)=\sum_{j\in\mathcal{R}_{i,o}\backslash\{i\}}f_{j}(x_{j})$, the dynamics of $p_{ii}$ described by $(\ref{NewCosDy})$ is consistent with the dynamics of the $i$th component of $\lambda$ described by (18). By solving the optimization problem (19), we know that the optimal control problem shares the same control rule (14) with the differential game problem. Since $p_{ii}(t)=\lambda_{i}(t)$ for every $t\in[0,T]$, we can see the statement in Corollary 2 holds.