A Passivity-Based Approach to Nash Equilibrium Seeking over Networks

TL;DR

This paper develops a passivity-based distributed algorithm for Nash equilibrium seeking over networks, allowing players with limited local information to converge to an equilibrium through local communication and augmented dynamics.

Contribution

It introduces a novel augmented gradient-play dynamics that enables NE seeking with partial information using passivity and Laplacian feedback over networks.

Findings

Convergence to NE under strict monotonicity of the pseudo-gradient.

Effective distributed dynamics based on local communication.

Tradeoffs between game properties and communication graph characteristics.

Abstract

In this paper we consider the problem of distributed Nash equilibrium (NE) seeking over networks, a setting in which players have limited local information. We start from a continuous-time gradient-play dynamics that converges to an NE under strict monotonicity of the pseudo-gradient and assumes perfect information, i.e., instantaneous all-to-all player communication. We consider how to modify this gradient-play dynamics in the case of partial, or networked information between players. We propose an augmented gradient-play dynamics with correction in which players communicate locally only with their neighbours to compute an estimate of the other players' actions. We derive the new dynamics based on the reformulation as a multi-agent coordination problem over an undirected graph. We exploit incremental passivity properties and show that a synchronizing, distributed Laplacian feedback can be designed using relative estimates of the neighbours. Under a strict monotonicity property of the pseudo-gradient, we show that the augmented gradient-play dynamics converges to consensus on the NE of the game. We further discuss two cases that highlight the tradeoff between properties of the game and the communication graph.

Equations

$${\Pi }_{\Omega }(x,v)=v-\beta (x)n^{\ast }(x)$$

$$v=v_{T_{\Omega }}+v_{N_{\Omega }}$$

$$\Sigma :\{\begin{array}{ll}\dot{x}=f(x,u),& \\ y=h(x,u),& \end{array}$$

$$\dot{V}(x)\le (y-\overline{y}{)}^T(u-\overline{u})$$

$$\dot{V}(x,x^{\prime })\le (y-y^{\prime }{)}^T(u-u^{\prime })$$

$$(\forall i\in \mathcal{I})(\forall y_i\in {\Omega }_i)J_i(x_i^{\ast },x_{-i}^{\ast })\le J_i(y_i,x_{-i}^{\ast })$$

$${\nabla }_i{J}_i(x_i^{\ast },x_{-i}^{\ast })=0,(\forall i\in \mathcal{I})\text{or}F(x^{\ast })=0$$

$$F(x)=[{\nabla }_1{J}_1^T(x),\dots ,{\nabla }_N{J}_N^T(x){]}^T$$

$$(x-x^{\ast }{)}^{T}F(x^{\ast })\ge 0\forall x\in \Omega$$

$$-F(x^{\ast })\in N_{\Omega }(x^{\ast })$$

$${\mathcal{P}}_i:$$

$${\tilde{\Sigma }}_i$$

$${\mathcal{R}}_i=[0_{n_i\times n_{<i}}{I}_{n_i}{0}_{n_i\times n_{>i}}]$$

$$\tilde{\Sigma }:\{\begin{array}{l}\dot{\mathbf{x}}=-{\mathcal{R}}^T\mathbf{F}(\mathbf{x})+B\mathbf{u}\\ \mathbf{y}=B^T\mathbf{x}\end{array}$$

$$\begin{array}{rl}\mathbf{F}(\mathbf{x})& =[{\nabla }_1{J}_1^T({\mathbf{x}}^1),\dots ,{\nabla }_N{J}_N^T({\mathbf{x}}^N){]}^T\end{array}$$

$$\dot{V}$$

$$=-(x-x^{\prime }{)}^T(\mathbf{F}(\mathbf{x})-\mathbf{F}({\mathbf{x}}^{\prime }))+(\mathbf{y}-{\mathbf{y}}^{\prime }{)}^T(\mathbf{u}-{\mathbf{u}}^{\prime })$$

$$\dot{V}\le (\mathbf{y}-{\mathbf{y}}^{\prime }{)}^T(\mathbf{u}-{\mathbf{u}}^{\prime })$$

$$Null(\mathbf{L})=Range(1_N\otimes I_n)$$

$$(1_N^T\otimes W)\mathbf{Lx}$$

$$\tilde{\mathcal{P}}:$$

$${\tilde{\mathcal{P}}}_i:$$

$${\tilde{\mathcal{P}}}_i:\{\begin{array}{l}\left[\begin{array}{c}{\dot{x}}_i\\ {\dot{\mathbf{x}}}_{-i}^i\end{array}\right]=\left[\begin{array}{c}-{\nabla }_i{J}_i(x_i,{\mathbf{x}}_{-i}^i)-{\mathcal{R}}_i{\sum }_{j\in {\mathcal{N}}_i}({\mathbf{x}}^i-{\mathbf{x}}^j)\\ -{\mathcal{S}}_i({\sum }_{j\in {\mathcal{N}}_i}{\mathbf{x}}^i-{\mathbf{x}}^j)\end{array}\right]\end{array}$$

$${\mathcal{S}}_i$$

$$0_{Nn}$$

$$0_n$$

$$\dot{V}=-(\mathbf{x}-\bar{\mathbf{x}}{)}^T{\mathcal{R}}^T(\mathbf{F}(\mathbf{x})-\mathbf{F}(\bar{\mathbf{x}}))-(\mathbf{x}-\bar{\mathbf{x}}{)}^T\mathbf{L}(\mathbf{x}-\bar{\mathbf{x}})$$

$$0$$

$$=-(x-x^{\ast }{)}^T[F(x)-F(x^{\ast })]<0\forall x\ne x^{\ast }$$

$$P_C=\frac{1}{N}{1}_N\otimes 1_N^T\otimes I_n,P_E=I_{Nn}-\frac{1}{N}{1}_N\otimes 1_N^T\otimes I_n$$

Full text

I Introduction

We consider distributed Nash equilibrium (NE) seeking over networks, where players have limited local information, over a communication network. This is a research topic of recent interest, [1], [2], [3], due to many networked scenarios in which such problems arise such as in wireless communication [4], [5], [6], optical networks [7], [8], [9], distributed constrained convex optimization [10], [11], noncooperative flow control problems [12], [13], etc.

We propose a new continuous-time dynamics for a general class of N-player games and prove its convergence to NE over a connected graph. Our scheme is derived based on reformulating the problem as a multi-agent coordination problem between the players and leveraging passivity properties. Specifically, we endow each player (agent) with an auxiliary state variable that provides an estimate of all other players’ actions. For each agent we combine its own gradient-type dynamics with an integrator-type auxiliary dynamics, driven by some control signal. We design the control signal for each individual player, based on the relative output feedback from its neighbours, such that these auxiliary state variables agree one with another. The resulting player’s dynamics has two components: the action component composed of a gradient term (enforcing the move towards minimizing its own cost) and a consensus term, and the estimate component with a consensus term. We call this new dynamics an augmented gradient-play dynamics with correction and estimation. We prove it converges to consensus on the NE of the game, under a monotonicity property of the extended pseudo-gradient.

Literature review. Our work is related to the literature of NE seeking in games over networks. Existing results are almost exclusively developed in discrete-time. The problem of NE seeking under networked communication is considered in [14],[15], specifically for the special class of aggregative games, where each agent’s cost function is coupled to other players’ actions through a single, aggregative variable. In [16], this approach is generalized to a larger class of coupled games: a gossip-based discrete-time algorithm is proposed and convergence was shown for diminishing step-sizes. Very recently, discrete-time ADMM-type algorithms with constant step-sizes have been proposed and convergence proved under co-coercivity of the extended pseudo-gradient, [17], [18].

In continuous-time, gradient-based dynamics for NE computation have been used since the work Arrow, [19], [20], [21], [22]. Over networks, gradient-based algorithms are designed in [11] based on information from only a set of local neighbouring agents for games with local utility functions (proved to be state-based potential games). Continuous-time distributed NE seeking dynamics are proposed for a two-network zero-sum game in [23]. Key assumptions are the additive decomposition of the common objective function as well as other structural assumptions. Based on the max-min formulation, the dynamics takes the form of a saddle-point dynamics, [19], distributed over the agents of each of the two networks, inspired by the optimization framework of [10].

In this paper we consider a general class of N-player games, where players have limited local information about the others’ actions over a communication network. Our work is also related to the distributed optimization framework in [10]. However there are several differences between [10] or [23] and our work. Beside the summable structure of the common cost function, in [10] a critical structural assumption is the fact that each agent optimizes its cost function over the full argument. Then, when an augmented (lifted) space of actions and estimates is considered in the networked communication case, a lifted cost function is obtained which can be decomposed as a sum of separable cost functions, individually convex in their full argument. This leads to distributed algorithms, under strict convexity of the individual cost functions with respect to the * full* argument. In a strategic game context, the individual convexity properties with respect to the full argument are too restrictive unless the game is separable to start with. While the game setting has an inherent distributed structure (since each player optimizes its own cost function), individual (player-by-player) optimization) is over its own action. In contrast to distributed optimization, each player’s individual action is only part of the full action profile and its cost function is coupled to its opponents’ actions, which are under their decision. This key differentiating structural aspect between games and distributed optimization presents technical challenges.

In this work, we also consider an augmented space to deal with the networked communication case, of actions and estimates of others’ actions. However, the difficulty is that we do not have an additive decomposition to exploit, and each player only controls/optimizes a part of the full argument on which its own cost function depends on. Our main approach is to highlight and exploit passivity properties of the game (pseudo-gradient), gradient-based algorithm/dynamics and network (Laplacian). A typical assumption in games is not individual gradient monotonicity with respect to the full argument, but rather monotonicity of the pseudo-gradient. The corresponding game assumption we use in the networked communication case, is monotonicity of the extended pseudo-gradient, which is equivalent to incremental passivity.

*Contributions. * We consider a general class of N-player games and develop of a new continuous-time dynamics for NE seeking dynamics under networked information. Our approach is based on reformulating the problem as a multi-agent coordination problem and exploiting basic incremental passivity properties of the pseudo-gradient map. To the best of our knowledge such an approach has not been proposed before. Our contributions are three-fold. First, we show that under strict monotonicity of the extended pseudo-gradient, the proposed new dynamics converges over any connected graph, unlike [17], [18]. Our scheme is different from [16], due to an extra correction term on the actions’ dynamics that arises naturally from the passivity-based design. This term is in fact critical to prove convergence on a single timescale. Essentially, players perform simultaneous consensus of estimates and player-by-player optimization.

Secondly, our passivity-based approach highlights the tradeoff between properties of the game and those of the communication graph. Under a weaker Lipschitz continuity assumption of the extended pseudo-gradient, we show that the new dynamics converges over any sufficiently connected graph. Key is the fact that the Laplacian contribution (or excess passivity) can be used to balance the other terms that are dependent on the game properties. Thirdly, we relax the connectivity bound on the graph, based on a time-scale separation argument. This is achieved by modifying the dynamics of the estimates such that the system approaches quickly the consensus subspace.

The paper is organized as follows. Section II gives the preliminary background. Section III formulates the noncooperative game and basic assumptions. Section IV presents the distributed NE seeking dynamics and analyzes its equilibrium points. Section V analyzes the convergence of the proposed dynamics over a connected graph under various assumptions. Section VI considers the case of compact action spaces, where projection dynamics are proposed and analyzed. Numerical examples are given in Section VII and conclusions in Section VIII.

II Preliminaries

Notations. Given a vector $x\in\mathbb{R}^{n}$, $x^{T}$ denotes its transpose. Let $x^{T}y$ denote the Euclidean inner product of $x,y\in\mathbb{R}^{n}$ and $\|x\|$ the Euclidean norm. Let $A\otimes B$ denote the Kronecker product of matrices $A$ and $B$. The all ones vector is $\mathbf{1}_{n}=[1,\dotsc,1]^{T}\in\mathbb{R}^{n}$, and the all zeros vector is $\mathbf{0}_{n}=[0,\dotsc,0]^{T}\in\mathbb{R}^{n}$. $diag(A_{1},\dotsc,A_{n})$ denotes the block-diagonal matrix with $A_{i}$ on its diagonal. Given a matrix $M\in\mathbb{R}^{p\times q}$, $Null(M)=\{x\in\mathbb{R}^{q}|Mx=0\}$ and $Range(M)=\{y\in\mathbb{R}^{p}|(\exists x\in\mathbb{R}^{q})\ y=Mx\}$. A function $\Phi:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$ is monotone if $(x-y)^{T}(\Phi(x)-\Phi(y))\geq 0$, for all $x,y\in\mathbb{R}^{n}$, and strictly monotone if the inequality is strict when $x\neq y$. $\Phi$ is strongly monotone if there exists $\mu>0$ such that $(x-y)^{T}(\Phi(x)-\Phi(y))\geq\mu\|x-y\|^{2}$, for all $x,y\in\mathbb{R}^{n}$. For a differentiable function $V:\mathbb{R}^{n}\rightarrow\mathbb{R}$, $\nabla V(x)=\frac{\partial V}{\partial x}(x)\in\mathbb{R}^{n}$ denotes its gradient. $V$ is convex, strictly, strongly convex if and only if its gradient $\nabla V$ is monotone, strictly, strongly monotone. Monotonicity properties play in variational inequalities the same role as convexity plays in optimization.

II-A Projections

Given a closed, convex set $\Omega\subset\mathbb{R}^{n}$, let the interior, boundary and closure of $\Omega$ be denoted by $\text{int}\Omega$, $\partial\Omega$ and $\overline{\Omega}$, respectively. The normal cone of $\Omega$ at a point $x\in{\Omega}$ is defined as $N_{\Omega}(x)=\{y\in\mathbb{R}^{n}|y^{T}(x^{\prime}-x)\leq 0,\forall x^{\prime}\in{\Omega}\}$. The tangent cone of $\Omega$ at $x\in{\Omega}$ is given as $T_{\Omega}(x)=\overline{\bigcup_{\delta>0}\frac{1}{\delta}(\Omega-x)}$. The projection operator of a point $x\in\mathbb{R}^{n}$ to the set $\Omega$ is given by the point $P_{\Omega}(x)\in\Omega$ such that $\left\lVert x-P_{\Omega}(x)\right\rVert\leq\left\lVert x-x^{\prime}\right\rVert$, for all $x^{\prime}\in\Omega$, or $P_{\Omega}(x)=\text{argmin}_{x^{\prime}\in\Omega}\left\lVert x-x^{\prime}\right\rVert$. The projection operator of a vector $v\in\mathbb{R}^{n}$ at a point $x\in\Omega$ with respect to $\Omega$ is $\Pi_{\Omega}(x,v)=\underset{\delta\to 0_{+}}{\lim}\frac{P_{\Omega}(x+\delta v)-x}{\delta}$. Note that $\left\lVert\Pi_{\Omega}(x,v)\right\rVert\leq\left\lVert v\right\rVert$. Given $x\in\partial\Omega$ let $n(x)$ denote the set of outward unit normals to $\Omega$ at $x$, $n(x)=\{y\,|\,y\in N_{\Omega}(x),\left\lVert y\right\rVert=1\}$. By Lemma 2.1 in [24], if $x\in\text{int}\Omega$, then $\Pi_{\Omega}(x,v)=v$, while if $x\in\partial\Omega$, then

[TABLE]

where $n^{*}(x)=\underset{n\in n(x)}{\text{argmax}}\,v^{T}\,n$ and $\beta(x)=\max\{0,v^{T}n^{*}(x)\}$. Note that if $v\in{T_{\Omega}(x)}$ for some $x\in\partial\Omega$, then $\underset{n\in n(x)}{\text{sup}}\,v^{T}\,n\leq 0$ hence $\beta(x)=0$ and no projection needs to be performed. The operator $\Pi_{\Omega}(x,v)$ is equivalent to the projection of the vector $v$ onto the tangent cone $T_{\Omega}(x)$ at $x$, $\Pi_{\Omega}(x,v)=P_{T_{\Omega}(x)}(v)$.

A set $C\subseteq\mathbb{R}^{n}$ is a cone if for any $c\in C$, $\gamma c\in C$ for every $\gamma>0$. The polar cone of a convex cone $C$ is given by $C^{\circ}=\{y\in\mathbb{R}^{n}|\,y^{T}c\leq 0,\ \forall c\in C\}$.

Lemma 1** (Moreau’s Decomposition Theorem III.3.2.5, [25]).**

Let $C\subseteq\mathbb{R}^{n}$ and $C^{\circ}\subseteq\mathbb{R}^{n}$ be a closed convex cone and its polar cone, and let $v\in\mathbb{R}^{n}$. Then the following are equivalent:

(i) $v_{C}=P_{C}(v)$ and $v_{C^{\circ}}=P_{C^{\circ}}(v)$.

(ii) $v_{C}\in C$, $v_{C^{\circ}}\in C^{\circ}$, $v=v_{C}+v_{C^{\circ}}$, and $v^{T}_{C}\,v_{C^{\circ}}=0$.

Notice that $N_{\Omega}(x)$ is a convex cone and the tangent cone is its polar cone, i.e., $N_{\Omega}(x)=(T_{\Omega}(x))^{\circ}$, $(N_{\Omega}(x))^{\circ}=T_{\Omega}(x)$. By Lemma 1, for any $x\in\Omega$, any vector $v\in\mathbb{R}^{n}$ can be decomposed into tangent $v_{T_{\Omega}}\in T_{\Omega}(x)$ and normal components, $v_{N_{\Omega}}\in N_{\Omega}(x)$,

[TABLE]

with $v_{T_{\Omega}}=P_{T_{\Omega}(x)}(v)=\Pi_{\Omega}(x,v)$, $v_{N_{\Omega}}=P_{N_{\Omega}(x)}(v)$.

II-B Graph theory

The following are from [26]. An undirected graph $G_{c}$ is a pair $G_{c}=(\mathcal{I},E)$ with $\mathcal{I}=\{1,\dotsc,N\}$ the vertex set and $E\subseteq\mathcal{I}\times\mathcal{I}$ the edge set such that for $i,j\in\mathcal{I}$, if $(i,j)\in E$, then $(j,i)\in E$. The degree of vertex $i$, $\text{deg}(i)$, is the number of edges connected to $i$. A path in a graph is a sequence of edges which connects a sequence of vertices. A graph is connected if there is a path between every pair of vertices. In this paper we associate a vertex with a player/agent. An edge between agents $i,j\in\mathcal{I}$ exists if agents $i$ and $j$ exchange information. Let $\mathcal{N}_{i}\subset\mathcal{I}$ denote the set of neighbours of player $i$. The Laplacian matrix $L\in\mathbb{R}^{N\times N}$ describes the connectivity of the graph $G_{c}$, with $[L]_{ij}=|\mathcal{N}_{i}|$, if $i=j$, $[L]_{ij}=-1$, if $j\in\mathcal{N}_{i}$ and [math] otherwise. When $G_{c}$ is an undirected and connected graph, [math] is a simple eigenvalue of $L$, $L\mathbf{1}_{N}=\mathbf{0}_{N}$, and all other eigenvalues positive. Let the eigenvalues of $L$ in ascending order be $0<\lambda_{2}\leq\dotsc\leq\lambda_{N}$, then $\underset{x\neq 0,\ \mathbf{1}_{N}^{T}x=0}{min}\,x^{T}Lx=\lambda_{2}\|x\|^{2}_{2}$, $\underset{x\neq 0}{max}\,x^{T}Lx=\lambda_{N}\|x\|_{2}^{2}$.

II-C Equilibrium Independent and Incremental Passivity

The following are from [27],[28],[29], [30]. Consider $\Sigma$

[TABLE]

with $x\in\mathbb{R}^{n}$, $u\in\mathbb{R}^{q}$ and $y\in\mathbb{R}^{q}$, $f$ locally Lipschitz and $h$ continuous. Consider a differentiable function $V:\mathbb{R}^{n}\rightarrow\mathbb{R}$. The time derivative of $V$ along solutions of (3) is denoted as $\dot{V}(x)=\nabla^{T}V(x)\,f(x,u)$ or just $\dot{V}$. Let $\overline{u}$, $\overline{x}$, $\overline{y}$ be an equilibrium condition, such that $0=f(\overline{x},\overline{u})$, $\overline{y}=h(\overline{x},\overline{u})$. Assume $\exists\overline{U}\subset\mathbb{R}$ and a continuous function $k_{x}(\overline{u})$ such that for any constant $\overline{u}\in\overline{U}$, $f(k_{x}(\overline{u}),\overline{u})=0$. The continuous function $k_{y}(\overline{u})=h(k_{x}(\overline{u}),\overline{u})$ is the equilibrium input-output map. Equilibrium independent passivity (EIP) requires $\Sigma$ to be passive independent of the equilibrium point.

Definition 1**.**

System $\Sigma$ (3) is Equilibrium Independent Passive (EIP) if it is passive with respect to $\overline{u}$ and $\overline{y}$; that is for every $\overline{u}\in\overline{U}$ there exists a differentiable, positive semi-definite storage function $V:\mathbb{R}^{n}\to\mathbb{R}$ such that $V(\overline{x})=0$ and, for all $u\in\mathbb{R}^{q}$, $x\in\mathbb{R}^{n}$,

[TABLE]

A slight refinement to the EIP definition can be made to handle the case where $k_{y}(\overline{u})$ is not a function but is a map. An EIP system with a map $k_{y}(\overline{u})$ is called maximal EIP (MEIP) when $k_{y}(\overline{u})$ is maximally monotone, e.g. an integrator, [28]. The parallel interconnection and the feedback interconnection of EIP systems results in a EIP system. When passivity holds in comparing any two trajectories of $\Sigma$, the property is called incremental passivity, [30].

Definition 2**.**

System $\Sigma$ (3) is incrementally passive if there exists a $C^{1}$, regular, positive semi-definite storage function $V:\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}$ such that for any two inputs $u$, $u^{\prime}$ and any two solutions $x$, $x^{\prime}$ corresponding to these inputs, the respective outputs $y$, $y^{\prime}$ satisfy

[TABLE]

where $\dot{V}=\nabla_{x}^{T}V(x,x^{\prime})f(x,u)+\nabla_{x^{\prime}}^{T}V(x,x^{\prime})f(x^{\prime},u^{\prime})$.

When $u^{\prime},x^{\prime},y^{\prime}$ are constant (equilibrium conditions), this recovers EIP definition. When system $\Sigma$ is just a static map, incrementally passivity reduces to monotonicity. A static function $y=\Phi(u)$ is EIP if and only if it is incrementally passive, or equivalently, it is monotone. Monotonicity plays an important role in optimization and variational inequalities while passivity plays as critical a role in dynamical systems.

III Problem Statement

III-A Game Formulation

Consider a set $\mathcal{I}=\{1,\dots,N\}$ of $N$ players (agents) involved in a game. The information sharing between them is described by an undirected graph $G_{c}=(\mathcal{I},E)$ or $G_{c}$.

Assumption 1**.**

The communication graph $G_{c}$ is connected.

Each player $i\in\mathcal{I}$ controls its action $x_{i}\in\Omega_{i}$, where $\Omega_{i}\subseteq\mathbb{R}^{n_{i}}$. The action set of all players is $\Omega=\prod_{i\in\mathcal{I}}\Omega_{i}\subseteq\mathbb{R}^{n}$, $n=\sum_{i\in\mathcal{I}}n_{i}$. Let $x=(x_{i},x_{-i})\in{\Omega}$ denote all agents’ action profile or $N$-tuple, where $x_{-i}\in\Omega_{-i}=\prod_{j\in\mathcal{I}\setminus\{i\}}\Omega_{j}$ is the $(N-1)$-tuple of all agents’ actions except agent $i$’s. Alternatively, $x$ is represented as a stacked vector $x=[x_{1}^{T}\dots x_{N}^{T}]^{T}\in\Omega\subseteq\mathbb{R}^{n}$. Each player (agent) $i$ aims to minimize its own cost function $J_{i}(x_{i},x_{-i})$, $J_{i}:\Omega\to\mathbb{R}$, which depends on possibly all other players’ actions. Let the game thus defined be denoted by $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$.

Definition 3**.**

Given a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$, an action profile $x^{*}=(x_{i}^{*},x_{-i}^{*})\in\Omega$ is a Nash Equilibrium (NE) of $\mathcal{G}$ if

[TABLE]

At a Nash Equilibrium no agent has any incentive to unilaterally deviate from its action. In the following we use one of the following two basic convexity and smoothness assumptions, which ensure the existence of a pure NE.

Assumption 2**.**

• (i)

For every $i\in\mathcal{I}$, $\Omega_{i}=\mathbb{R}^{n_{i}}$, the cost function $J_{i}:\Omega\to\mathbb{R}$ is $\mathcal{C}^{2}$ in its arguments, strictly convex and radially unbounded in $x_{i}$, for every $x_{-i}\in{\Omega}_{-i}$.

• (ii)

For every $i\in\mathcal{I}$, $\Omega_{i}$ is a non empty, convex and compact subset of $\mathbb{R}^{n_{i}}$ and the cost function $J_{i}:\Omega\to\mathbb{R}$ is $\mathcal{C}^{1}$ in its arguments and (strictly) convex in $x_{i}$, for every $x_{-i}\in{\Omega}_{-i}$.

Under Assumption 2(i) from Corollary 4.2 in [31] it follows that a pure NE $x^{*}$ exists. Moreover, an NE satisfies

[TABLE]

where $\nabla_{i}J_{i}(x_{i},x_{-i})=\frac{\partial J_{i}}{\partial x_{i}}(x_{i},x_{-i})\in\mathbb{R}^{n_{i}}$, is the gradient of agent $i$’s cost function $J_{i}(x_{i},x_{-i})$ with respect to its own action $x_{i}$ and $F:\Omega\to\mathbb{R}^{n}$ is the pseudo-gradient defined by stacking all agents’ partial gradients,

[TABLE]

Under Assumption 2(ii) it follows from Theorem 4.3 in [31] that a pure NE exists, based on Brower’s fixed point theorem. Under just convexity of $J_{i}$ with respect to $x_{i}$, existence of an NE follows based on a Kakutani’s fixed point theorem. Moreover a Nash equilibrium (NE) $x^{*}\in\Omega$ satisfies the variational inequality (VI) (cf. Proposition 1.4.2, [32]),

[TABLE]

and projected gradient methods need be used, [32]. Additionally (6) can be written as $-F(x^{*})^{T}(x-x^{*})\leq 0$ and from the definition of the normal cone,

[TABLE]

Next we state typical assumptions on the pseudo-gradient.

Assumption 3**.**

• (i)

The pseudo-gradient $F:\Omega\to\mathbb{R}^{n}$ is strictly monotone, $(x-x^{\prime})^{T}(F(x)-F(x^{\prime}))>0,\,\forall x\neq x^{\prime}$.

• (ii)

The pseudo-gradient $F:\Omega\to\mathbb{R}^{n}$ is strongly monotone, $(x-x^{\prime})^{T}(F(x)-F(x^{\prime}))\geq\mu\|x-x^{\prime}\|^{2},\,\forall x,x^{\prime}\in\Omega$, for $\mu>0$, and Lipschitz continuous, $\|F(x)-F(x^{\prime})\|\leq\underline{\theta}\|x-x^{\prime}\|,\,\forall x,x^{\prime}\in\Omega$, where $\underline{\theta}>0$.

Under Assumption 3(i) or 3(ii), the game has a unique NE, (cf. Theorem 3 in [33]).

The above setting refers to players’ strategic interactions, but it does not specify what knowledge or information each player has. Since $J_{i}$ depends on all players’ actions, an introspective calculation of an NE requires complete information where each player knows the cost functions and strategies of all other players, see Definition 3 and (4). A game with incomplete information refers to players not fully knowing the cost functions or strategies of the others, [34]. Throughout the paper, we assume $J_{i}$ is known by player $i$ only. In a game with incomplete but perfect information, each agent has knowledge of the actions of all other players, $x_{-i}$. We refer to the case when players are not able to observe the actions of all the other players, as a game with incomplete and imperfect or partial information. This is the setting we consider in this paper: we assume players can communicate only locally, with their neighbours. Our goal is to derive a dynamics for seeking an NE in the incomplete, partial information case, over a communication graph $G_{c}$. We review first the case of perfect information, and treat the case of partial or networked information in the following sections. In the first part of the paper, for simplicity of the arguments, we consider $\Omega_{i}=\mathbb{R}^{n_{i}}$ and Assumption 2(i). We consider compact action sets and treat the case of projected dynamics, under Assumption 2(ii), in Section VI.

III-B Gradient Dynamics with Perfect information

In a game of perfect information, under Assumption 2(i), a typical gradient-based dynamics, [20], [21], [22], [32], can be used for each player $i$

[TABLE]

or, $\mathcal{P}:\,\,\,\dot{x}=-F(x)$, the overall system of all the agents’ dynamics stacked together. Assumption 2(i) ensures existence and uniqueness of solutions of (8). Note that (8) requires all-to-all instantaneous information exchange between players or over a complete communication graph. Convergence of (8) is typically shown under strict (strong) monotonicity on the pseudo-gradient $F$, [32], [21], or under strict diagonal dominance of its Jacobian evaluated at $x^{*}$,[1]. We provide a passivity interpretation. The gradient dynamics $\mathcal{P}$ (8) is the feedback interconnection between a bank of $N$ integrators $\Sigma$ and the static pseudo-gradient map $F(\cdot)$, Figure 1.

$\Sigma$ is MEIP with storage function $V(x)=\frac{1}{2}\|x-\overline{x}\|^{2}$, while $F(\cdot)$ is static and under Assumption 3(i) is incrementally passive (EIP). Hence their interconnection is also EIP and asymptotic stability can be shown using the same storage function.

Lemma 2**.**

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ in the perfect information case, under Assumptions 2(i). Then, the equilibrium point $\bar{x}$ of the gradient dynamics, (8) is the NE of the game $x^{*}$ and, under Assumption 3(i), is globally asymptotically. Alternatively, under 3(ii) $x^{*}$ is exponentially stable, hence the solutions of (8) converge exponentially to the NE of the game, $x^{*}$.

Proof.

At an equilibrium $\overline{x}$, of (8), $F(\overline{x})=0$, hence by (4), $\bar{x}=x^{*}$, the NE of $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$. Consider the quadratic Lyapunov function $V:\Omega\to\mathbb{R}$, $V(x)=\frac{1}{2}\|x-\bar{x}\|^{2}$. Along (8) using $F(\overline{x})=0$, $\dot{V}(x)=-(x-\overline{x})^{T}(F(x)-F(\overline{x}))<0$, for all $x\neq\overline{x}$, by Assumption 3(i). Hence $\dot{V}(x)\leq 0$ and $\dot{V}(x)=0$ only if $x=\overline{x}=x^{*}$. Since $V$ is radially unbounded, the conclusion follows by LaSalle’s theorem [35]. Under Assumption 3(ii), $\dot{V}(x)\leq-\mu\|x-\overline{x}\|^{2}$, $\forall x$ and global exponential stability follows immediately. ∎

IV NE Seeking Dynamics over a Graph

In this section we consider the following question: how can we modify the gradient dynamics (8) such that it converges to NE in a networked information setting, over some connected communication graph $G_{c}$?

We propose a new augmented gradient dynamics, derived based on the reformulation as a multi-agent agreement problem between the players. We endow each player (agent) with an auxiliary state that provides an estimate of all other players’ actions. We design a new signal for each player, based on the relative feedback from its neighbours, such that these estimates agree one with another.

Thus assume that player (agent) $i$ maintains an estimate vector $\mathbf{x}^{i}=[(\mathbf{x}_{1}^{i})^{T},\dotsc,(\mathbf{x}_{N}^{i})^{T}]^{T}\in\Omega$ where $\mathbf{x}_{j}^{i}$ is player $i$’s estimate of player $j$’s action and $\mathbf{x}_{i}^{i}=x_{i}$ is player $i$’s actual action. $\mathbf{x}_{-i}^{i}$ represents player $i$’s estimate vector without its own action, $\mathbf{x}_{i}^{i}$. All agents’ vectors are stacked into a single vector $\mathbf{x}=[(\mathbf{x}^{1})^{T},\dotsc,(\mathbf{x}^{N})^{T}]^{T}\in\prod_{i\in\mathcal{I}}\Omega=\Omega^{N}=\mathbb{R}^{Nn}$. Note that the state space is now $\Omega^{N}=\prod_{i\in\mathcal{I}}\Omega=\mathbb{R}^{Nn}$. In the enlarged space the estimate components will be different initially, but in the limit all players estimate vectors should be in consensus. We modify the gradient dynamics such that player $i$ updates $\mathbf{x}_{i}^{i}$ to reduce its own cost function and updates $\mathbf{x}^{i}_{-i}$ to reach a consensus with the other players. Let each player combine its gradient-type dynamics with an integrator-type auxiliary dynamics, driven by some control signal,

[TABLE]

where $B^{i}$ is a full rank ${n\times n}$ matrix. For each player, $\mathbf{u}_{i}\in\mathbb{R}^{n}$ is to be designed based on the relative output feedback from its neighbours, such that $\mathbf{x}^{i}=\mathbf{x}^{j}$, for all $i,j$, and converge to the NE $x^{*}$.

Thus we have reformulated the design of NE dynamics over $G_{c}$ as a multi-agent agreement problem. We note that agent dynamics (9) are heterogenous, separable, but do not satisfy an individual passivity property as typically assumed in multi-agent literature, e.g. [29], [36]. We show next that a Laplacian-type feedback can be designed under strict incremental passivity of the pseudo-gradient.

To proceed, we first analyze properties of $\widetilde{\Sigma}_{i}$ and the overall agents’ dynamics $\widetilde{\Sigma}$. Write $\widetilde{\Sigma}_{i}$ (9) in a compact form

[TABLE]

where

[TABLE]

and $n_{<i}=\sum_{j<i\ i,j\in\mathcal{I}}n_{j}$, $n_{>i}=\sum_{j>i\ i,j\in\mathcal{I}}n_{j}$.

Thus $\mathcal{R}_{i}^{T}\in\mathbb{R}^{n\times n_{i}}$ aligns the gradient to the action component in $\dot{\mathbf{x}}_{i}$. From (10), with $\mathbf{x}=[(\mathbf{x}^{1})^{T},\dotsc,(\mathbf{x}^{N})^{T}]^{T}$, $\mathbf{u}=[\mathbf{u}^{T}_{1},\dots,\mathbf{u}^{T}_{N}]^{T}\in\mathbb{R}^{Nn}$, the overall agents’ dynamics denoted by $\widetilde{\Sigma}$ can be written in stacked form as

[TABLE]

where $\mathcal{R}=diag(\mathcal{R}_{1},\dots,\mathcal{R}_{N})$, $B=diag(B^{1},\dots,B^{N})$ and

[TABLE]

is the continuous extension of the pseudo-gradient $F$, (5) to the augmented space, $\mathbf{F}(\mathbf{x}):\Omega^{N}\to\mathbb{R}^{n}$. Note that $\mathbf{F}(\mathbf{1}_{N}\otimes x)=F(x)$. In the rest of the paper we consider one of the following two assumptions on the extended $\mathbf{F}$.

Assumption 4**.**

• (i)

The extended pseudo-gradient $\mathbf{F}$ is monotone, $(x-x^{\prime})^{T}(\mathbf{F}(\mathbf{x})-\mathbf{F}(\mathbf{x}^{\prime}))\geq 0,\,\forall\mathbf{x},\mathbf{x}^{\prime}\in\Omega^{N}$.

• (ii)

*The extended pseudo-gradient $\mathbf{F}$ is Lipschitz continuous, *$\|\mathbf{F}(\mathbf{x})-\mathbf{F}(\mathbf{x}^{\prime})\|\leq\theta\|\mathbf{x}-\mathbf{x}^{\prime}\|,\,\forall\mathbf{x},\mathbf{x}^{\prime}\in\Omega^{N}$where $\theta>0$.

Remark 1**.**

We compare this assumption to similar ones used in distributed optimization and in multi-agent coordination control, respectively. First, note that Assumption 4(i) on the extended pseudo-gradient $\mathbf{F}$ holds under individual joint convexity of each $J_{i}$ with respect to the full argument. In distributed optimization problems, each objective function is assumed to be strictly (strongly) jointly convex in the full vector $\mathbf{x}$ and its gradient to be Lipschitz continuous, e.g. [10]. Similarly, in multi-agent coordination control, it is standard to assume that individual agent dynamics are separable and strictly (strongly) incrementally passive, e.g. [29]. However, in a game context the individual joint convexity of $J_{i}$ with respect to the full argument is too restrictive, unless we have a trivial game with separable cost functions. In general, $J_{i}$ is coupled to other players’ actions while each player has under its control only its own action. This is a key difference versus distributed optimization or multi-agent coordination, one which introduces technical challenges. However, we show that under the monotonicity Assumption 4(i) on $\mathbf{F}$, the overall $\widetilde{\Sigma}$ (12) is incrementally passive, hence EIP. Based on this, we design a new dynamics which converges over any connected $G_{c}$ (Theorem 1). Under the weaker Lipschitz Assumption 4(ii) on $\mathbf{F}$ and Assumption 3(ii) on $F$, we show that the new dynamics converges over any sufficiently connected $G_{c}$ (Theorem 2). We also note that Assumption 4(i) is similar to those used in [17], [18], while Assumption 4(ii) is weaker. Assumption 4(i) is the extension of Assumption 3(i) to the augmented space, for local communication over the connected graph $G_{c}$. The weaker Assumption 4(ii) on $\mathbf{F}$, is the extension of Lipschitz continuity of $F$ in Assumption 3(ii). We also note that these assumptions could be relaxed to hold only locally around $x^{*}$ in which case all results become local.

Lemma 3**.**

Under Assumption 4(i), the overall system $\widetilde{\Sigma}$, (12), is incrementally passive, hence EIP.

Proof.

Consider two inputs $\mathbf{u}$, $\mathbf{u}^{\prime}$ and let $\mathbf{x}$, $\mathbf{x}^{\prime}$, $\mathbf{y}$, $\mathbf{y}^{\prime}$ be the trajectories and outputs of $\widetilde{\Sigma}$ (12). Let the storage function be $V(\mathbf{x},\mathbf{x}^{\prime})=\frac{1}{2}\|\mathbf{x}-\mathbf{x}^{\prime}\|^{2}$. Then, along solutions of (12),

[TABLE]

by using $\mathcal{R}=diag(\mathcal{R}_{1},\dots,\mathcal{R}_{N})$ and (11). Using Assumption 4(i) it follows that

[TABLE]

Thus by Definition 2, $\widetilde{\Sigma}$, is incrementally passive, hence EIP. ∎

IV-A Distributed feedback design

Given agent dynamics $\widetilde{\Sigma}_{i}$, (10), for each individual player we design $\mathbf{u}_{i}\in\mathbb{R}^{n}$ based on the relative output feedback from its neighbours, such that the auxiliary state variables (estimates) agree one with another and converge to the NE $x^{*}$. For simplicity, take $B=I_{Nn}$ so that $\mathbf{y}=\mathbf{x}$.

Let $\mathcal{N}_{i}$ denote the set of neighbours of player $i$ in graph $G_{c}$ and $L$ denote the symmetric Laplacian matrix. Let $\mathbf{L}=L\otimes I_{n}$ denote the augmented Laplacian matrix which satisfies

[TABLE]

and $Range(\mathbf{L})=Null(\mathbf{1}_{N}^{T}\otimes I_{n})$, based on $L\mathbf{1}_{N}=\mathbf{0}_{N}$. For any $W\in\mathbb{R}^{q\times n}$, and any $\mathbf{x}\in\mathbb{R}^{Nn}$, using $L\mathbf{1}_{N}=\mathbf{0}_{N}$,

[TABLE]

With respect to the overall dynamics $\widetilde{\Sigma}$, (12), the objective is to design $\mathbf{u}$ such that $\mathbf{x}$ reaches consensus, i.e., $\mathbf{1}_{N}\otimes x$, for some $x\in\Omega$ and $x$ converges towards the NE $x^{*}$. The consensus condition is written as $\mathbf{L}\mathbf{x}=0$. Since $\widetilde{\Sigma}$, (12) is incrementally passive by Lemma 3, and $\mathbf{L}$ is positive semi-definite, a passivity-based control design, e.g. [36], suggests taking $\mathbf{u}=-\mathbf{L}\mathbf{x}$. The resulting closed-loop system which represents the new overall system dynamics $\widetilde{\mathcal{P}}$ is given in stacked notation as

[TABLE]

shown in Figure 2 as the feedback interconnection between $\widetilde{\Sigma}$ and $\mathbf{L}$. Local solutions of (17) exist by Assumption 2(i).

The new individual player dynamics $\widetilde{\mathcal{P}}_{i}$ are

[TABLE]

or, separating the action $\mathbf{x}^{i}_{i}=x_{i}$ and estimate $\mathbf{x}^{i}_{-i}$ dynamics,

[TABLE]

where

[TABLE]

and $\mathcal{S}_{i}$ removes $\mathbf{x}^{i}_{i}=x_{i}$, its own action component from agent $i$’s estimate vector, $\mathbf{x}^{i}$.

For player $i$, $\widetilde{\mathcal{P}}_{i}$, (18) or (19) is clearly distributed over $G_{c}$. Its input is the relative difference between its estimate and its neighbours’. In standard consensus terms, agent $i$ can use this information to move in the direction of the average value of its neighbours, while the gradient term enforces the move towards minimizing its own cost. Compared to the gossip-based algorithm in [16], the action part of (19) has an extra correction term. This term is instrumental in proving convergence on a single timescale as shown in Section V.

The next result shows that the equilibrium of (17) or (19) occurs when the agents are at a consensus and at NE.

Lemma 4**.**

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ over a communication graph $G_{c}$ under Assumptions 1, 2(i). Let the dynamics for each agent $\widetilde{\mathcal{P}}_{i}$ be as in (18), (19), or overall $\widetilde{\mathcal{P}}$, (17). At an equilibrium point $\overline{\mathbf{x}}\in{\Omega}^{N}$ the estimate vectors of all players are equal $\bar{\mathbf{x}}^{i}=\bar{\mathbf{x}}^{j}$, $\forall i,j\in\mathcal{I}$ and equal to the Nash equilibrium profile $x^{*}$, hence the action components of all players coincide with the optimal actions, $\bar{\mathbf{x}}^{i}_{i}=x^{*}_{i}$, $\forall i\in\mathcal{I}$.

Proof.

Let $\overline{\mathbf{x}}$ denote an equilibrium of (17),

[TABLE]

Pre-multiplying both sides by $(\mathbf{1}_{N}^{T}\otimes I_{n})$, yields $\mathbf{0}_{n}=-(\mathbf{1}_{N}^{T}\otimes I_{n})\mathcal{R}^{T}\mathbf{F}(\bar{\mathbf{x}})$, where $(\mathbf{1}_{N}^{T}\otimes I_{n})\mathbf{L}\bar{\mathbf{x}}=0$ was used by (15). Using (11) and simplifying $(\mathbf{1}_{N}^{T}\otimes I_{n})\mathcal{R}^{T}$ gives

[TABLE]

by (13). Substituting (22) into (21) results in $\mathbf{0}_{Nn}=-\mathbf{L}\bar{\mathbf{x}}$. From this it follows that $\bar{\mathbf{x}}^{i}=\bar{\mathbf{x}}^{j}$, $\forall i,j\in\mathcal{I}$ by Assumption 1 and (15). Therefore $\bar{\mathbf{x}}=\mathbf{1}_{N}\otimes\bar{x}$, for some $\bar{x}\in{\Omega}$. Substituting this back into (22) yields $\mathbf{0}_{n}=\mathbf{F}(\mathbf{1}_{N}\otimes\bar{x})$ or $\nabla_{i}J_{i}(\bar{x})=0$, for all $i\in\mathcal{I}$. Using (13), $\nabla_{i}J_{i}(\bar{x}_{i},\bar{x}_{-i})=0$, for all $i\in\mathcal{I}$, or $\mathbf{0}_{n}=F(\bar{x})$. Therefore by (4) $\bar{x}=x^{*}$, hence $\bar{\mathbf{x}}=\mathbf{1}_{N}\otimes x^{*}$ and for all $i,j\in\mathcal{I}$, $\bar{\mathbf{x}}^{i}=\bar{\mathbf{x}}^{j}=x^{*}$ the NE of the game. ∎

V Convergence Analysis

In this section we analyze the convergence of player’s new dynamics $\widetilde{\mathcal{P}}_{i}$ (18), (19) or overall $\widetilde{\mathcal{P}}$ (17) to the NE of the game, over a connected graph $G_{c}$. We consider two cases.

In Section V-A we analyze convergence of (19) on a single timescale: in Theorem 1 under Assumptions 1, 2(i), 3(i) and 4(i), and in Theorem 2 under Assumptions 1, 2(i), 3(ii) and 4(ii). We exploit the incremental passivity (EIP) property of $\widetilde{\Sigma}$ (12) (Lemma 3) and diffusive properties of the Laplacian.

In Section V-B, we modify the estimate component of the dynamics (19) to be much faster, and in Theorem 3 prove convergence under Assumptions 1, 2(i), 3(ii) and 4(ii), using a two-timescale singular perturbation approach.

V-A Single-Timescale Consensus and Player Optimization

Theorem 1 shows that, under Assumption 4(i), (19) converges to the NE of the game, over any connected $G_{c}$.

Theorem 1**.**

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ over a communication graph $G_{c}$ under Assumptions 1, 2(i), 3(i) and 4(i). Let each player’s dynamics $\widetilde{\mathcal{P}}_{i}$, be as in (18), (19), or overall $\widetilde{\mathcal{P}}$, (17), as in Figure 2. Then, any solution of (17) is bounded and asymptotically converges to $\mathbf{1}_{N}\otimes x^{*}$, and the actions components converge to the NE of the game, $x^{*}$.

Proof.

By Lemma 4, the equilibrium of $\widetilde{\mathcal{P}}$ (17) is $\bar{\mathbf{x}}=\mathbf{1}_{N}\otimes x^{*}$. We consider the quadratic storage function $V(\mathbf{x})=\frac{1}{2}\|\mathbf{x}-\bar{\mathbf{x}}\|^{2}$, $\bar{\mathbf{x}}=\mathbf{1}_{N}\otimes x^{*}$ as a Lyapunov function. As in (IV) in the proof of Lemma 3, using $\mathbf{u}=-\mathbf{L}\mathbf{x}$, (21) we obtain that along the solutions of (17),

[TABLE]

where $\bar{\mathbf{x}}=\mathbf{1}_{N}\otimes x^{*}$, and $\mathbf{x}^{T}\mathcal{R}^{T}=x$. By Assumption 4(i) and since the augmented Laplacian $\mathbf{L}$ is positive semi-definite it follows that $\dot{V}\leq 0$, for all $\mathbf{x}\in{\Omega}^{N}$, hence all trajectories of (17) are bounded and $\overline{\mathbf{x}}$ is stable. To show convergence we resort to LaSalle’s invariance principle, [35].

From (23), $\dot{V}=0$ when both terms in (23) are zero, i.e., $(\mathbf{x}-\bar{\mathbf{x}})^{T}\mathcal{R}^{T}(\mathbf{F}(\mathbf{x})-\mathbf{F}(\bar{\mathbf{x}}))=0$ and $(\mathbf{x}-\bar{\mathbf{x}})^{T}\mathbf{L}(\mathbf{x}-\bar{\mathbf{x}})=0$. By Assumption 1 and (15), $(\mathbf{x}-\bar{\mathbf{x}})^{T}\mathbf{L}(\mathbf{x}-\bar{\mathbf{x}})=0$ is equivalent to $\mathbf{x}-\bar{\mathbf{x}}=\mathbf{1}_{N}\otimes\underline{x}$, for some $\underline{x}\in\mathbb{R}^{n}$. Since at equilibrium $\bar{\mathbf{x}}=\mathbf{1}_{N}\otimes x^{*}$, this implies that $\mathbf{x}=\mathbf{1}_{N}\otimes x$, for some $x\in\mathbb{R}^{n}$. By (13) $\mathbf{F}(\mathbf{1}_{N}\otimes x)=F(x)$. Using $\mathcal{R}$ in (11) yields for the first term in (23) to be zero,

[TABLE]

where strict inequality follows by Assumption 3(i). Therefore $\dot{V}=0$ in (23) only if $x=x^{*}$ and hence $\mathbf{x}=\mathbf{1}_{N}\otimes x^{*}$. Since $V$ is radially unbounded, the conclusion follows by LaSalle’s invariance principle. ∎

If $\mathbf{F}(\cdot)$ is strongly monotone, exponential convergence can be shown over any connected $G_{c}$. Next we show that, under a weaker Lipschitz property of $\mathbf{F}$ (Assumption 4(ii)) and strong monotonicity of $F$ (Assumption 3(ii)), (19) converges over any sufficiently connected $G_{c}$.

Theorem 2**.**

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ over a communication graph $G_{c}$ under Assumptions 1, 2(i), 3(ii) and 4(ii). Let each player’s dynamics $\widetilde{\mathcal{P}}_{i}$ be as in (18), (19) or overall $\widetilde{\mathcal{P}}$ (17). Then, if $\lambda_{2}(L)>\frac{\theta^{2}}{\mu}+\theta$, any solution of (17) converges asymptotically to $\mathbf{1}_{N}\otimes x^{*}$, and the actions components converge to the NE of the game, $x^{*}$. If $\lambda_{2}(L)>\frac{N\theta^{2}}{\mu}+\theta$, then convergence is exponential.

Proof.

We decompose $\mathbb{R}^{Nn}$ as $\mathbb{R}^{Nn}=C_{N}^{n}\oplus E_{N}^{n}$, into the consensus subspace $C_{N}^{n}=\{\mathbf{1}_{N}\otimes x\,|x\in\mathbb{R}^{n}\}$ and its orthogonal complement $E_{N}^{n}$. Let two projection matrices be defined as

[TABLE]

Then any $\mathbf{x}\in\mathbb{R}^{Nn}$ can be decomposed as $\mathbf{x}=\mathbf{x}^{||}\,+\,\mathbf{x}^{\perp}$, where $\mathbf{x}^{||}=P_{C}\mathbf{x}\in C_{N}^{n}$ and $\mathbf{x}^{\perp}=P_{E}\mathbf{x}\in E_{N}^{n}$, with $(\mathbf{x}^{||})^{T}\mathbf{x}^{\perp}=0$. Thus $\mathbf{x}^{||}=\mathbf{1}_{N}\otimes x$, for some $x\in\mathbb{R}^{n}$, so that $\mathbf{L}\mathbf{x}^{||}=0$, and $min_{\mathbf{x}^{\perp}\in E_{N}^{n}}(\mathbf{x}^{\perp})^{T}\mathbf{L}\mathbf{x}^{\perp}=\lambda_{2}(L)\|\mathbf{x}^{\perp}\|^{2}$, $\lambda_{2}(L)>0$.

Consider $V(\mathbf{x})=\frac{1}{2}\|\mathbf{x}-\overline{\mathbf{x}}\|^{2}$, $\overline{\mathbf{x}}=\mathbf{1}_{N}\otimes x^{*}$, which using $\mathbf{x}=\mathbf{x}^{||}\,+\,\mathbf{x}^{\perp}$ can be written as $V(\mathbf{x})=\frac{1}{2}\|\mathbf{x}^{\perp}\|^{2}+\frac{1}{2}\|\mathbf{x}^{||}-\overline{\mathbf{x}}\|^{2}$.

Then following the same steps as in Lemma 3, and replacing $\mathbf{x}$ with its decomposed components $\mathbf{x}^{\perp},\mathbf{x}^{||}$ a relation similar to (23) follows along (17), i.e.,

[TABLE]

Using $\mathbf{x}=\mathbf{x}^{\perp}+\mathbf{x}^{||}$, $\overline{\mathbf{x}}=\mathbf{1}_{N}\otimes x^{*}$, $\mathbf{L}\mathbf{x}^{||}=0$, $\dot{V}$ can be written as

[TABLE]

Using $(\mathbf{x}^{\perp})^{T}\mathbf{L}\mathbf{x}^{\perp}\geq\lambda_{2}(L)\|\mathbf{x}^{\perp}\|^{2}$ and $\mathbf{F}(\mathbf{x}^{||})=F(x)$, $\mathbf{F}(\overline{\mathbf{x}})=F(x^{*})$ yields

[TABLE]

Under Assumption 4(ii), $\|\mathbf{F}(\mathbf{x})-\mathbf{F}(\mathbf{x}^{||})\|\leq\theta\|\mathbf{x}^{\perp}\|$, so that, after simplifying $\mathcal{R}\mathbf{x}^{||}=x$, $\mathcal{R}(\mathbf{1}_{N}\otimes x^{*})=x^{*}$,

[TABLE]

Using again Assumption 4(ii) in the $3^{rd}$ and $2^{nd}$ terms and Assumption 3(ii) in the $4^{th}$ one, it can be shown that

[TABLE]

or $\dot{V}\leq-[\|x-x^{*}\|\,\,\,\,\,\|\mathbf{x}^{\perp}\|]\,\Theta\,[\|x-x^{*}\|\,\,\,\,\,\|\mathbf{x}^{\perp}\|]^{T}$, where \Theta=\left[\begin{array}[]{cc}\mu&-\theta\\ -\theta&\lambda_{2}(L)-\theta\end{array}\right]. Under the conditions in the statement, $\Theta$ is positive definite. Hence $\dot{V}\leq 0$ and $\dot{V}=0$ only if $\mathbf{x}^{\perp}=0$ and $x=x^{*}$, hence $\mathbf{x}=\mathbf{1}_{N}\otimes x^{*}$. The conclusion follows by LaSalle’s invariance principle.

Exponential convergence follows using $\|x-x^{*}\|=\frac{1}{\sqrt{N}}\|\mathbf{x}^{||}-\overline{\mathbf{x}}\|$, under the stricter condition on $\lambda_{2}(L)$. Indeed,

[TABLE]

where \Theta_{N}=\left[\begin{array}[]{cc}\frac{1}{N}\mu&-\theta\\ -\theta&\lambda_{2}(L)-\theta\end{array}\right] is positive definite if $\lambda_{2}(L)>\frac{N\theta^{2}}{\mu}+\theta$. This implies that $\dot{V}(\mathbf{x}(t))\leq-\eta V(\mathbf{x}(t))$, for some $\eta>0$, so that $\mathbf{x}(t)$ converges exponentially to $\overline{\mathbf{x}}$. ∎

Remark 2**.**

Since $\theta,\mu$ are related to the coupling in the players’ cost functions and $\lambda_{2}(L)$ to the connectivity between players, Theorem 2 highlights the tradeoff between properties of the game and those of the communication graph $G_{c}$. Key is the fact that the Laplacian contribution can be used to balance the other terms in $\dot{V}$. Alternatively $\mathbf{L}$ on feedback path in Figure 2 has excess passivity which compensates the lack of passivity in the $\mathbf{F}$ terms, or $\widetilde{\Sigma}$ on the forward path .

Remark 3**.**

We note that we can relax the monotonicity assumption to hold just at the NE $x^{*}$, recovering a strict-diagonal assumption used in [1]. However, since $x^{*}$ is unknown, such an assumption cannot be checked a-priori except for special cases such as quadratic games, (see Section VII). Local results follow if assumptions for $F(\cdot)$ hold only locally around $x^{*}$, and for $\mathbf{F}(\cdot)$ only locally around $\mathbf{x}^{*}=\mathbf{1}_{N}\otimes x^{*}$ . We note that the class of quadratic games satisfies Assumption 3(ii) globally.

An alternative representation of the dynamics $\widetilde{\mathcal{P}}$, (17), reveals interesting connections to distributed optimization and passivity based control, [10], [36]. To do that we use two matrices to write a compact representation for $\widetilde{\mathcal{P}}$, (17). Let $\mathcal{S}=diag(\mathcal{S}_{1},...,\mathcal{S}_{N})\in\mathbb{R}^{(Nn-n)\times Nn}$, where $\mathcal{S}_{i}$ in (20). Then $\mathcal{S}$ and $\mathcal{R}$ (11) satisfy $\mathcal{S}\mathcal{R}^{T}=\mathbf{0}$ and

[TABLE]

Using $\mathcal{R}$ and $\mathcal{S}$, the stacked actions are $[(\mathbf{x}^{1}_{1})^{T},\dotsc,(\mathbf{x}^{N}_{N})^{T}]^{T}=\mathcal{R}\mathbf{x}$, while the stacked estimates $[(\mathbf{x}^{1}_{-1})^{T},\dotsc,(\mathbf{x}^{N}_{-N})^{T}]^{T}=\mathcal{S}\mathbf{x}$. Let $x=\mathcal{R}\mathbf{x}$ and $z=\mathcal{S}\mathbf{x}$, and using properties of $\mathcal{R}$, $\mathcal{S}$, (26), yields $\mathbf{x}=\mathcal{R}^{T}x+\mathcal{S}^{T}z$. Thus the equivalent representation of $\widetilde{\mathcal{P}}$ (17) is:

[TABLE]

which separates the actions and the estimates stacked components of the dynamics. Using $\mathbf{L}=L\otimes I_{n}$ and $L=QQ^{T}$, with $Q$ the incidence matrix, yields the interconnected block-diagram in Figure 3.

We note that, unlike distributed optimization (e.g. [10]) and passivity based control (e.g. [36]) where the dynamics are decoupled, in Figure 2 the dynamics on the top path are coupled, due to the inherent coupling in players’ cost functions in game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$. As another observation, recall that given a matrix $Q$, pre-multiplication by $Q$ and post-multiplication by $Q^{T}$ preserves passivity of a system. Figure 2 shows that possible generalized dynamics can be designed by substituting the identity block on the feedback path by some other passive dynamics. Based on Figure 2, one such dynamic generalization can be obtained by substituting the static feedback through $\mathbf{L}$, with an integrator or proportional-integrator term through $\mathbf{L}$, which preserves passivity, as in [10]. Thus the passivity interpretation of the game NE seeking dynamics design allows a systematic derivation of new dynamics/algorithms.

Note that if the dynamics of the estimates were modified such that the system approached quickly the consensus subspace then convergence to the NE could be shown via a time-scale decomposition approach. This is explored in the next section.

V-B Two-Timescale Singular Perturbation Analysis

In this section we relax the connectivity bound on $L$ in Theorem 2, based on a time-scale separation argument. The idea is to modify the dynamics of the estimates in $\widetilde{\mathcal{P}}$ (17) or (27) such that the system approaches quickly the consensus subspace. Under Assumption 3(ii) and 4(ii), we show convergence to the NE over a sufficiently connected $G_{c}$ based on a time-scale decomposition approach.

Recall the equivalent representation of $\widetilde{\mathcal{P}}$ in (27) and modify the estimate component of the dynamics such that is much faster than the action component,

[TABLE]

where $\epsilon>0$. Thus player $i$’s dynamics is as follows:

[TABLE]

with the $\frac{1}{\epsilon}$ high gain on the estimate component. $\widetilde{\mathcal{P}}_{\epsilon}$ (28) is in the standard form of a singularly perturbed system, where the estimate dynamics and the action dynamics are the fast and the slow components, respectively.

Theorem 3**.**

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ over a communication graph $G_{c}$ under Assumptions 1, 2(i), 3(ii) and 4(ii). Let each player’s dynamics $\widetilde{\mathcal{P}}_{i,\epsilon}$ be as in (29), or overall $\widetilde{\mathcal{P}}_{\epsilon}$ (28), $\epsilon>0$. Then, there exists $\epsilon^{*}>0$, such that for all $0<\epsilon<\epsilon^{*}$, $(x^{*},\mathcal{S}(\mathbf{1}_{N}\otimes x^{*}))$ is exponentially stable. Alternatively, $(x^{*},\mathcal{S}(\mathbf{1}_{N}\otimes x^{*}))$ is asymptotically stable, for all $0<\epsilon<1$ such that

[TABLE]

Proof.

We analyze (28) by examining the reduced and the boundary-layer systems. First we find the roots of $\mathcal{S}\mathbf{L}[\mathcal{R}^{T}x+\mathcal{S}^{T}z]=0$, or $\mathcal{S}\mathbf{L}\mathbf{x}=0$. Note that, by (26), $\mathbf{x}\in Null(\mathcal{S}\mathbf{L})$ if and only if $\mathbf{L}\mathbf{x}\in Null(\mathcal{S})$, which is equivalent to $\mathbf{L}\mathbf{x}\in Range(\mathcal{R^{T}})$. Thus $\mathbf{x}\in Null(\mathcal{S}\mathbf{L})$ if and only if there exists $q\in\mathbb{R}^{n}$ such that $\mathbf{L}\mathbf{x}=\mathcal{R^{T}}q$. Then for such $q\in\mathbb{R}^{n}$ and for all $w\in\mathbb{R}^{n}$, $(\mathbf{1}^{T}_{N}\otimes w^{T})\mathbf{L}\mathbf{x}=(\mathbf{1}^{T}_{N}\otimes w^{T})\mathcal{R}^{T}q$. Using $(\mathbf{1}^{T}_{N}\otimes w^{T})\mathbf{L}\mathbf{x}=0$ by (16) and $\mathcal{R}$ in (11), this means $0=w^{T}q$, for all $w\in\mathbb{R}^{n}$. Therefore, $q=0$ and $\mathbf{L}\mathbf{x}=0$. By (15), $\mathbf{x}=\mathbf{1}_{N}\otimes x$, $x\in\Omega$. Hence roots of $\mathcal{S}\mathbf{L}[\mathcal{R}^{T}x+\mathcal{S}^{T}z]=0$ are when $\mathbf{x}=(\mathbf{1}_{N}\otimes x)$, i.e., $z=\mathcal{S}(\mathbf{1}_{N}\otimes x)$.

We use a change of coordinates $v=z-\mathcal{S}(\mathbf{1}_{N}\otimes x)$, to shift the equilibrium of the boundary-layer system to the origin. First, we use $z=v+\mathcal{S}(\mathbf{1}_{N}\otimes x)$ and $x=\mathcal{R}(\mathbf{1}_{N}\otimes x)$ to rewrite the term $\mathcal{R}^{T}x+\mathcal{S}^{T}z$ that appears in (28) as follows,

[TABLE]

where (26) was used. Using this and the change of variables $v=z-\mathcal{S}(\mathbf{1}_{N}\otimes x)$ into (28) with $\mathbf{L}(\mathbf{1}_{N}\otimes x)=0$ yields,

[TABLE]

Note that $v=0$ is the quasi-steady state of $\epsilon\dot{v}$ and substituting this in $\dot{x}$ gives the reduced system as

[TABLE]

which is exactly the gradient dynamics and has equilibrium $x^{*}$at the NE. By Lemma 2, under Assumption 3(ii) the gradient dynamics, (31), is exponentially stable.

The boundary-layer system on the $\tau=t/\epsilon$ timescale is

[TABLE]

It can be shown that matrix $\mathcal{S}\mathbf{L}\mathcal{S}^{T}$ is positive definite, so that (32) is exponentially stable. To see this note that $\mathcal{S}\mathbf{L}\mathcal{S}^{T}v=0$ only if $\mathcal{S}^{T}v\in Null(\mathcal{S}\mathbf{L})$. Recall that $Null(\mathcal{S}\mathbf{L})=Null(\mathbf{L})=(\mathbf{1}_{N}\otimes w)$, $w\in{\Omega}$. Note that to be in $Null(\mathbf{L})$, $y=[(y^{1})^{T}\dotsc(y^{N})^{T}]^{T}=\mathcal{S}^{T}v$ has to have $y^{i}=y^{j}$, $\forall i,j\in\mathcal{I}$, but $y$ has a [math] in component $y^{i}_{i}\ \forall i\in\mathcal{I}$, due to the definition of $\mathcal{S}$. Therefore $y^{i}\neq y^{j}$ unless $y^{j}_{i}=y^{i}_{i}=0$, for all $j$. Therefore $y=\mathcal{S}^{T}v$ has to be equal to [math] to be in the $Null(\mathbf{L})$. Since $Null(\mathcal{S}^{T})=\{0\}$ this implies $v=0$, hence $Null(\mathcal{S}\mathbf{L}\mathcal{S}^{T})=0$ and $\mathcal{S}\mathbf{L}\mathcal{S}^{T}$ is positive definite. By Theorem 11.4 in [35] it follows that there exists $\epsilon^{*}>0$, such that for all $\epsilon<\epsilon^{*}$, $(x^{*},0)$ is exponentially stable for (30), or $(x^{*},\mathcal{S}(\mathbf{1}_{N}\otimes x^{*}))$ is exponentially stable for (28).

Alternatively, Theorem 11.3 in [35] can be applied to (30) to show asymptotic stability. The two Lyapunov functions are $V(x)=\frac{1}{2}\|x-x^{*}\|^{2}$ and $W(v)=\frac{1}{2}\|v\|^{2}$, and along the reduced and the boundary layer-systems, (31),(32), the following hold

[TABLE]

[TABLE]

so that (11.39) and (11.40) in [35] hold for $\alpha_{1}=\mu$, $\psi_{1}(x)=\|x-x^{*}\|$, and $\alpha_{2}=\lambda_{2}(L)/N$, $\psi_{2}(v)=\|v\|$.

Note also that the following holds

[TABLE]

so that (11.43) in [35] holds for $\beta_{1}=\theta+\lambda_{N}(L)$.

Similarly,

[TABLE]

and (11.44), [35] holds for $\beta_{2}=\sqrt{N}\theta$, $\gamma=\sqrt{N}(\theta+\lambda_{N}(L))$.

Then using Theorem 11.3, $\epsilon^{*}=\frac{\alpha_{1}\alpha_{2}}{\alpha_{1}\gamma+\beta_{1}\beta_{2}}$, is given as

[TABLE]

and using $\lambda_{N}(L)\leq 2d^{*}=2\max_{i\in\mathcal{I}}|\mathcal{N}_{i}|$,

[TABLE]

Then, by Theorem 11.3 in [35], for any $0<\epsilon<1$ such that

[TABLE]

$(x^{*},0)$ is asymptotic stable for (30), hence $(x^{*},\mathcal{S}(\mathbf{1}_{N}\otimes x^{*}))$ is asymptotic stable for (28).

∎

Remark 4**.**

In $\widetilde{\mathcal{P}}_{i,\epsilon}$ (29) the estimate dynamics is made faster with the gain $1/\epsilon$. It can be shown that for sufficiently high $1/\epsilon$, $\frac{1}{\epsilon}>N\sqrt{N}(1+2\frac{d^{*}}{\theta})$, the bound on $\lambda_{2}(L)$ in Theorem 3 is lower than the bound on $\lambda_{2}(L)$ in Theorem 2. Alternatively, we can consider a gain parameter $\frac{1}{\epsilon}>0$ on the estimates in (17) to improve the lower bound to $\lambda_{2}(L)>\epsilon(\frac{\theta^{2}}{\mu}+\theta)$, as shown in the next section. Thus a higher $1/\epsilon$ can relax the connectivity bound on $L$, but $\epsilon$ is a global parameter. This highlights another aspect of the tradeoff between game properties (coupling), communication graph (consensus) properties and information.

VI Projected NE Dynamics for Compact Action Sets

In this section we treat the case of compact $\Omega_{i}$ action sets, under Assumption 2(ii), using projected dynamics. We highlight the major steps of the approach and their differences compared to the unconstrained action set case.

In a game of perfect information, for compact $\Omega_{i}$ action set, under Assumption 2(ii) each player $i\in\mathcal{I}$ runs the projected gradient-based dynamics, given as [21], [22],

[TABLE]

The overall system of all agents’ projected dynamics in stacked notation is given by

[TABLE]

or, equivalently,

[TABLE]

where equivalence follows by using Lemma 1 (Moreau’s decomposition theorem, [25]), or directly by Proposition 1 and Corollary 1 in [37]. Furthermore, this is equivalent to the differential inclusion [38]

[TABLE]

In all the above the projection operator is discontinuous on the boundary of $\Omega$. We use the standard definition of a solution of a projected dynamical system (PDS), (Definition 2.5 in [24]). Thus we call $x:[0,+\infty)\rightarrow\Omega$ a solution of (34) if $x(\cdot)$ is an absolutely continuous function $t\mapsto x(t)$ and $\dot{x}(t)=\Pi_{\Omega}\left(x(t),-F(x(t))\right)$ holds almost everywhere (a.e.) with respect to $t$, i.e., except on a set of measure zero.

The existence of a unique solution of (34) is guaranteed for any $x(0)\in\Omega$, under Lipschitz continuity of $F$ on $\Omega$, cf. Theorem 2.5 in [24]. Note that any solution must necessarily lie in $\Omega$ for almost every $t$. Alternatively, existence holds under continuity and (hypo) monotonicity of $F$, i.e., for some $\mu\leq 0$,

[TABLE]

(see Assumption 2.1 in [24], and also Theorem 1 in [37] for extension to a non-autonomous systems). This is similar to the QUAD relaxation in [39]. It means that $F(x)-\mu x$ is monotone, where $\mu\leq 0$. Note that $F(x)+\eta x$ is strongly monotone for any $\eta>-\mu$, with $\eta+\mu>0$ monotonicity constant. When $\mu>0$ in fact we can take $\eta=0$ and recover $\mu$-strong monotonicity of $F$. Thus under Assumption 2(ii), 3(i), for any $x(0)\in\Omega$, there exists a unique solution of (34) and moreover, a.e., $\dot{x}(t)\in T_{\Omega}(x(t))$ and $x(t)\in\Omega$.

Equilibrium points of (34) coincide with Nash equilibria, which are solutions of the VI($F$, $\Omega$), by Theorem 2.4 in [24]. To see this, let $\overline{x}$ be an equilibrium point of (34) such that $\{\overline{x}\in\Omega\,|\,\mathbf{0}_{n}=\Pi_{\Omega}\left(\overline{x},-F(\overline{x})\right)\}$. By Lemma 2.1 in [24] and (1), if $\overline{x}\in\text{int}\Omega$, then $\mathbf{0}_{n}=\Pi_{\Omega}\left(\overline{x},-F(\overline{x})\right)=-F(\overline{x})$, while if $\overline{x}\in\partial\Omega$, then

[TABLE]

for some $\beta>0$ and $n\in n(\overline{x})\subset N_{\Omega}(\overline{x})$. Equivalently, by (35), $-F(\overline{x})\in N_{\Omega}(\overline{x})$ and using the definition of $N_{\Omega}(\overline{x})$, it follows that

[TABLE]

Comparing to (6), or (7), it follows that $\overline{x}=x^{*}$. Thus the equilibrium points of (34) $\{x^{*}\in\Omega\,|\,\mathbf{0}_{n}=\Pi_{\Omega}(x^{*},-F(x^{*}))\}$ coincide with Nash equilibria $x^{*}$.

Lemma 5**.**

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ in the perfect information case, under Assumptions 2(ii) and 3(i). Then, for any $x_{i}(0)\in\Omega_{i}$, the solution of (33), $i\in\mathcal{I}$, or (34) converges asymptotically to the NE of the game $x^{*}$. Under Assumption 3(ii) convergence is exponential.

Proof.

The proof follows from Theorem 3.6 and 3.7 in [24]. Consider any $x(0)\in\Omega$ and $V(t,x)=\frac{1}{2}\|x(t)-x^{*}\|^{2}$, where $x^{*}$ is the Nash equilibrium of the game, and $x(t)$ is the solution of (34). Then the time derivative of $V$ along solutions of (34) is $\dot{V}=(x(t)-x^{*})^{T}\Pi_{\Omega}(x(t),-F(x(t)))$. Since $T_{\Omega}(x(t))=[N_{\Omega}(x(t))]^{o}$, by Moreau’s decomposition theorem (Lemma 1), at any point $x(t)\in\Omega$ the pseudo-gradient $-F(x(t))$ can be decomposed as in (2) into normal and tangent components, in $N_{\Omega}(x(t))$ and $T_{\Omega}(x(t))$. Since $\Pi_{\Omega}(x(t),-F(x(t)))=P_{T_{\Omega}(x(t))}(-F(x(t)))$ is in the tangent cone $T_{\Omega}(x(t))$, it follows as in (35) that

[TABLE]

From the definition of the normal cone $N_{\Omega}(x(t))$ this means,

[TABLE]

for all $x^{\prime}\in{\Omega}$. Thus it follows that for $x^{\prime}=x^{*}$ and $\forall x(t)\in\Omega$,

[TABLE]

From (6), at the Nash equilibrium $F(x^{*})^{T}(x(t)-x^{*})\geq 0$, $\forall x(t)\in\Omega$. Therefore adding this to the right-hand side of the above and using $\dot{V}(t)$ yields that along solutions of (34), for all $t\geq 0$, $\dot{V}=(x(t)-x^{*})^{T}\Pi_{\Omega}(x(t),-F(x(t)))\leq-(x(t)-x^{*})^{T}(F(x(t))-F(x^{*}))<0$, when $x(t)\neq x^{*}$, where the strict inequality follows from Assumption 3(i). Hence $V(t)$ is monotonically decreasing and non-negative, and thus there exists $\lim_{t\rightarrow\infty}V(t)=\underline{V}$. As in Theorem 3.6 in [24], a contradiction argument can be used to show that $\underline{V}=0$, hence for any $x(0)\in\Omega$, $\|x(t)-x^{*}\|\rightarrow 0$ as $t\rightarrow\infty$.

Under Assumption 3(ii), for any $x(0)\in\Omega$, along solutions of (34), for all $t\geq 0$, $\dot{V}\leq-\mu\|x(t)-x^{*}\|^{2}=-\mu V(t)$, $\mu>0$, $\forall x$ and exponential convergence follows immediately. ∎

In the partial or networked information case, over graph $G_{c}$, we modify each player’s $\widetilde{\Sigma}_{i}$, in (9) using projected dynamics for the action components to $\Omega_{i}$, as in

[TABLE]

where $\mathbf{u}_{i}(t)\in\mathbb{R}^{n}$ is a piecewise continuous function, to be designed based on the relative output feedback from its neighbours, such that $\mathbf{x}^{i}=\mathbf{x}^{j}$, for all $i,j$, and converge towards the NE $x^{*}$. Write $\widetilde{\Sigma}_{i}$ (38) in a more compact form

[TABLE]

where $\mathcal{R}_{i}$, $\mathcal{S}_{i}$, are defined as in (11), (20). The overall dynamics for all players becomes in stacked form, $\widetilde{\Sigma}$,

[TABLE]

where $x=\mathcal{R}\mathbf{x}$, $\mathcal{R}=diag(\mathcal{R}_{1},\dots,\mathcal{R}_{N})$, $\mathcal{S}=diag(\mathcal{S}_{1},\dots,\mathcal{S}_{N})$, satisfying the properties in (26), and $\mathbf{u}(t)\in\mathbb{R}^{Nn}$ is piecewise continuous. This is similar to (12), except that the dynamics for the action components is projected to $\Omega$. For any $\mathcal{R}\mathbf{x}(0)\in{\Omega}$, existence of a unique solution of (40) is guaranteed under Assumption 4(i) or (ii) by Theorem 1 in [37]. Extending the incrementally passivity and EIP concept to projected dynamical systems leads to the following result.

Lemma 6**.**

Under Assumption 4(i), the overall system $\widetilde{\Sigma}$, (40), is incrementally passive, hence EIP.

Proof.

Consider two inputs $\mathbf{u}(t)$, $\mathbf{u}^{\prime}(t)$ and let $\mathbf{x}(t)$, $\mathbf{x}^{\prime}(t)$, $\mathbf{y}(t)$, $\mathbf{y}^{\prime}(t)$ be the state trajectories and outputs of $\widetilde{\Sigma}$ (40). Let the storage function be $V(t,\mathbf{x},\mathbf{x}^{\prime})=\frac{1}{2}\|\mathbf{x}(t)-\mathbf{x}^{\prime}(t)\|^{2}$. Then, along solutions of (40),

[TABLE]

Notice that using (26) the following holds for (40),

[TABLE]

Hence as in (37), for any $\mathcal{R}\mathbf{x}(t)\in{\Omega}$,

[TABLE]

and using the definition of normal cone $N_{\Omega}(\mathcal{R}\mathbf{x}(t))$,

[TABLE]

for all $\mathcal{R}\mathbf{x}^{\prime}(t)\in{\Omega}$. Since $\mathcal{R}\mathbf{x}(t)\in{\Omega}$ and $\mathcal{R}\mathbf{x}^{\prime}(t)\in{\Omega}$ are both arbitrary elements in $\Omega$, swapping them leads to

[TABLE]

Adding (42) and (43) results in

[TABLE]

Therefore using this in (VI), yields for $\dot{V}$

[TABLE]

or, using $\mathcal{R}^{T}\mathcal{R}+\mathcal{S}^{T}\mathcal{S}=I$,

[TABLE]

Finally, using Assumption 4(i), it follows that

[TABLE]

and $\widetilde{\Sigma}$ is incrementally passive, hence EIP. ∎

Since $\widetilde{\Sigma}$, (40) is incrementally passive by Lemma 6, and $\mathbf{L}$ is positive semi-definite, as in Section IV-A we consider a passivity-based control $\mathbf{u}(t)=-\mathbf{L}\mathbf{x}(t)$. The resulting closed-loop system which represents the new overall system dynamics $\widetilde{\mathcal{P}}$ is given in stacked notation as

[TABLE]

Alternatively, using $x=\mathcal{R}\mathbf{x}$, $z=\mathcal{S}\mathbf{x}$, $\mathcal{R}\mathcal{S}^{T}=0$, equivalently with actions and estimates separated as in (27),

[TABLE]

Existence of a unique solution of (45) or (46) is guaranteed under Assumption 4(i) or (ii) by Theorem 1 in [37]. From (45) or (46) after separating the action $\mathbf{x}^{i}_{i}=x_{i}$ and estimate $\mathbf{x}^{i}_{-i}$ dynamics, the new projected player dynamics $\widetilde{\mathcal{P}}_{i}$ are,

[TABLE]

Compared to (19), $\widetilde{\mathcal{P}}_{i}$, in (47) has projected action components. The next result shows that the equilibrium of (45) or (47) occurs when the agents are at a consensus and at NE.

Lemma 7**.**

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ over a communication graph $G_{c}$ under Assumptions 1, 2(ii) and 4(i) or (ii). Let the dynamics for each agent $\widetilde{\mathcal{P}}_{i}$ be as in (47), or overall $\widetilde{\mathcal{P}}$, (45). At an equilibrium point $\overline{\mathbf{x}}$ the estimate vectors of all players are equal $\bar{\mathbf{x}}^{i}=\bar{\mathbf{x}}^{j}$, $\forall i,j\in\mathcal{I}$ and equal to the Nash equilibrium profile $x^{*}$, hence the action components of all players coincide with the optimal actions, $\bar{\mathbf{x}}^{i}_{i}=x^{*}_{i}$, $\forall i\in\mathcal{I}$.

Proof.

Let $\overline{\mathbf{x}}$ denote an equilibrium of (45),

[TABLE]

Pre-multiplying both sides by $\mathcal{R}$ and using (26) simplifies to,

[TABLE]

Substituting (49) into (48) results in $\mathbf{0}_{Nn}=-\mathcal{S}^{T}\mathcal{S}\mathbf{L}\bar{\mathbf{x}}$ which implies $\bar{\mathbf{x}}\in null(\mathbf{L})$. From this it follows that $\bar{\mathbf{x}}^{i}=\bar{\mathbf{x}}^{j}$, $\forall i,j\in\mathcal{I}$ by Assumption 1 and (15). Therefore $\bar{\mathbf{x}}=\mathbf{1}_{N}\otimes\bar{x}$, for some $\bar{x}\in{\Omega}$. Substituting this back into (49) yields $\mathbf{0}_{n}=\Pi_{\Omega}\left(\mathcal{R}(\mathbf{1}_{N}\otimes\bar{x}),-\mathbf{F}(\mathbf{1}_{N}\otimes\bar{x})\right)$ or $\mathbf{0}_{n}=\Pi_{\Omega}(\bar{x},-F(\bar{x}))$ by using (13). Therefore as in (36) it follows that $-F(\bar{x})\in N_{\Omega}(\bar{x})$ hence, by (7), $\bar{x}=x^{*}$ the NE. Thus $\bar{\mathbf{x}}=\mathbf{1}_{N}\otimes x^{*}$ and for all $i,j\in\mathcal{I}$, $\bar{\mathbf{x}}^{i}=\bar{\mathbf{x}}^{j}=x^{*}$ the NE of the game. ∎

The following results show single-timescale convergence to the NE of the game over a connected $G_{c}$, under Assumption 4(i) or Assumption 4(ii).

Theorem 4**.**

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ over a communication graph $G_{c}$ under Assumptions 1, 2(ii), 3(i) and 4(i). Let each player’s dynamics $\widetilde{\mathcal{P}}_{i}$, be as in (47), or overall $\widetilde{\mathcal{P}}$, (45). Then, for any $x(0)\in\Omega$ and any $z(0)$, the solution of (45) asymptotically converges to $\mathbf{1}_{N}\otimes x^{*}$, and the actions components converge to the NE of the game, $x^{*}$.

Proof.

The proof is similar to the proof of Theorem 1 except that, instead of LaSalle’s invariance principle, the argument is based on Barbalat’s Lemma, [35], since the system is time-varying. Let $V(t,\mathbf{x})=\frac{1}{2}\|\mathbf{x}(t)-\overline{\mathbf{x}}\|^{2}$, where by Lemma 7, $\overline{\mathbf{x}}=\mathbf{1}_{N}\otimes x^{*}$. Using (VI) in Lemma 6, for $\mathbf{x^{\prime}}(t)=\overline{\mathbf{x}}$, $\mathbf{u}(t)=-\mathbf{L}\mathbf{x}(t)$, $\mathbf{u^{\prime}}(t)=-\mathbf{L}\overline{\mathbf{x}}$, it follows that for any $x(0)\in\Omega$ and any $z(0)$, along (45),

[TABLE]

Under Assumption 4(i), $\dot{V}\leq 0$, for all $\mathcal{R}\mathbf{x}(t)\in\Omega$, $z(0)$. Thus $V(t,\mathbf{x}(t))$ is non-increasing and bounded from below by [math], hence it converges as $t\rightarrow\infty$ to some $\underline{V}\geq 0$. Then, under Assumption 4(i), it follows that $\lim_{t\rightarrow\infty}\int_{0}^{t}(\mathbf{x}(\tau)-\bar{\mathbf{x}})^{T}\mathbf{L}(\mathbf{x}(\tau)-\bar{\mathbf{x}})d\tau$ exists and is finite. Since $\mathbf{x}(t)$ is absolutely continuous, hence uniformly continuous, from Barbalat’s Lemma in [35] it follows that $\mathbf{L}(\mathbf{x}(t)-\bar{\mathbf{x}})\rightarrow 0$ as $t\rightarrow\infty$. Since $\overline{\mathbf{x}}=\mathbf{1}_{N}\otimes x^{*}$, this means that $\mathbf{x}(t)\rightarrow\mathbf{1}_{N}\otimes x$, as $t\rightarrow\infty$ for some $x\in\Omega$. Then $V(t,\mathbf{x}(t))=\frac{1}{2}\|\mathbf{x}(t)-\overline{\mathbf{x}}\|^{2}\rightarrow\frac{1}{2}\|\mathbf{1}_{N}\otimes(x-x^{*})\|^{2}=\underline{V}$ as $t\rightarrow\infty$. If $\underline{V}=0$ the proof if completed. Using the strict monotonicity assumption 3(i), it can be shown by a contradiction argument that $x=x^{*}$ and $\underline{V}=0$. Assume that $x\neq x^{*}$ and $\underline{V}>0$. Then from (50) there exists a sequence $\{t_{k}\},t_{k}\rightarrow\infty$, as $k\rightarrow\infty$, such that $\dot{V}(t_{k})\rightarrow 0$ as $k\rightarrow\infty$. Suppose this claim is false. Then there exists a $d>0$ and a $T>0$ such that $\dot{V}(t)\leq-d$, for all $t>T$, which contradicts $\underline{V}>0$, hence the claim is true. Substituting $\{t_{k}\}$ into (50) yields

[TABLE]

where the left-hand side converges to [math] as $k\rightarrow\infty$. Hence,

[TABLE]

Using $\lim_{k\rightarrow\infty}\mathbf{x}(t_{k})=\mathbf{1}_{N}\otimes x\in Null(\mathbf{L})$, this leads to

[TABLE]

or, $0\leq-(x-x^{*})]^{T}(F(x)-F(x^{*}))<0$, by the strict monotonicity Assumption 3(i), since we assumed $x\neq x^{*}$. This is a contradiction, hence $x=x^{*}$ and $\underline{V}=0$. ∎

Theorem 5**.**

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ over a communication graph $G_{c}$ under Assumptions 1, 2(ii), 3(ii) and 4(ii). Let each player’s dynamics $\widetilde{\mathcal{P}}_{i}$ be as in (47) or overall $\widetilde{\mathcal{P}}$ (45). Then, if $\lambda_{2}(L)>\frac{\theta^{2}}{\mu}+\theta$, for any $x(0)\in\Omega$ and any $z(0)$, the solution of (45) asymptotically converges to $\mathbf{1}_{N}\otimes x^{*}$, and the actions converge to the NE of the game, $x^{*}$. If $\lambda_{2}(L)>\frac{N\theta^{2}}{\mu}+\theta$, then convergence is exponential.

Proof.

The proof is similar to the proof of Theorem 2. Based on Lemma 7, using $V(t,\mathbf{x},\overline{\mathbf{x}})=\frac{1}{2}\|\mathbf{x}(t)-\overline{\mathbf{x}}\|^{2}$ and (VI) in Lemma 6, for $\mathbf{u}(t)=-\mathbf{L}\mathbf{x}(t)$, $\mathbf{u^{\prime}}(t)=-\mathbf{L}\overline{\mathbf{x}}(t)$ one can obtain (50) along (45), for any $x(0)\in\Omega$ and any $z(0)$. Then further decomposing $\mathbf{x}(t)$ into $\mathbf{x}^{\perp}(t)$ and $\mathbf{x}^{||}(t)$ components as in the proof of Theorem 2 leads to an inequality as (25), where $\Theta$ is positive under the conditions in the theorem. Then invoking Barbalat’s Lemma in [35] as in the proof of Theorem 4 leads to $x(t)\rightarrow x^{*}$ and $\mathbf{x}^{\perp}(t)\rightarrow 0$ as $t\rightarrow\infty$. ∎

Note also that we can consider a gain parameter $\frac{1}{\epsilon}>0$ on the estimates in (45) to improve the lower bound to $\lambda_{2}(L)$. Consider

[TABLE]

Thus player $i$’s dynamics is as follows:

[TABLE]

Following the proof of Theorem 2 the matrix $\Theta$ becomes

[TABLE]

where the condition for the matrix to be positive definite is $\lambda_{2}(L)>\epsilon(\frac{\theta^{2}}{\mu}+\theta)$. As in the two-time scale analysis this $\epsilon$ is a global parameter.

VII Numerical Examples

VII-A Unconstrained $\Omega$ and Dynamics

Example 1: Consider a N-player quadratic game from economics, where 20 firms are involved in the production of a homogeneous commodity. The quantity produced by firm $i$ is denoted by $x_{i}$. The overall cost function of firm $i$ is $J_{i}(x_{i},x_{-i})=c_{i}(x_{i})-x_{i}f(x)$, where $c_{i}(x_{i})=(20+10(i-1))x_{i}$ is the production cost, $f(x)=2200-\sum_{i\in\mathcal{I}}x_{i}$ is the demand price, as in [18]. We investigate the proposed dynamics (19) over a communication graph $G_{c}$ via simulation. The initial conditions are selected randomly from $[0,20]$. Assumption 3(i) and 4(i) hold, so by Theorem 1 the dynamics (19) will converge even over a minimally connected graph. Figures 7 and 7 show the convergence of (19) over a randomly generated communication graph $G_{c}$ (Fig. 5) and over a cycle $G_{c}$ graph (Fig. 5), respectively.

Example 2: Consider a second example of an 8 player game with $J_{i}(x_{i},x_{-i})=c_{i}(x_{i})-x_{i}f(x)$, $c_{i}(x_{i})=(10+4(i-1))x_{i}$, $f(x)=600-\sum_{i\in\mathcal{I}}x_{i}^{2}$, as in [16]. Here Assumption 4(i) on $\mathbf{F}$ does not hold globally, so cannot apply Theorem 1, but Assumption 4(ii) holds locally. By Theorem 2, (19) will converge depending on $\lambda_{2}(L)$. Figure 11 shows the convergence of (19) over a sufficiently connected, randomly generated communication graph $G_{c}$ as depicted in Fig. 9. Over a cycle $G_{c}$ graph, (19) does not converge. Alternatively, by Theorem 3, a higher $1/\epsilon$ (time-scale decomposition) can balance the connectivity loss. Fig. 11 shows convergence for (29) with $1/\epsilon=200$, over a cycle $G_{c}$ graph as shown in Fig. 9. The initial conditions are selected randomly from $[0,20]$.

VII-B Compact $\Omega$ and Projected Dynamics

Example 1: $N=20$ and this time $\Omega_{i}=[0,200]$. We investigate the projected augmented gradient dynamics (47) over a graph $G_{c}$. The actions’ initial conditions are selected randomly from $[0,200]$, while the estimates from $[-2000,2000]$. Assumption 3(i) and 4(i) hold, so by Theorem 4 the dynamics (47) will converge even over a minimally connected graph. Figures 15 and 15 show the convergence of (47) over a randomly generated communication graph $G_{c}$ (Fig. 13) and over a cycle $G_{c}$ graph (Fig. 13), respectively.

*Example 2: * $N=8$ and $\Omega_{i}=[0,20]$, as in [16]. Here Assumption 4(i) on $\mathbf{F}$ does not hold globally, so cannot apply Theorem 4. Under Assumption 3(ii) and 4(ii) by Theorem 2, (47) will converge depending on $\lambda_{2}(L)$. Figure 19 shows the convergence of (47) over a sufficiently connected, randomly generated communication graph $G_{c}$ as in Fig. 17. A higher $1/\epsilon$ on the estimates can balance the lack of connectivity. Fig. 19 shows results for (52) with $1/\epsilon=200$, over a cycle $G_{c}$ graph as in Fig. 17. The actions’ initial conditions are selected randomly from $[0,20]$ and the estimates from $[-200,200]$.

Example 3: Consider $J_{i}(x_{i},x_{-i})=c_{i}(x_{i})-x_{i}f(x)$, where $c_{i}(x_{i})=[20+40(i-1)]x_{i}$ and $f(x)=1200-\sum_{i}(x_{i})$, $\Omega_{i}=[0,200]$, for $N=20$. The NE is on the boundary $\begin{bmatrix}200&200&183.3&143.3&103.3&63.3&23.3&0&\cdots\end{bmatrix}$. Figure 23 shows the convergence of (47) over a randomly generated communication graph $G_{c}$ as in Fig. 21, while Fig. 23 gives similar results, this time over a cycle $G_{c}$ graph as depicted in Fig. 21.

VIII Conclusion

In this paper, we studied distributed Nash equilibrium (NE) seeking over networks in continuous-time. We proposed an augmented gradient-play dynamics with estimation in which players communicate locally only with their neighbours to compute an estimate of the other players’ actions. We derived the new dynamics based on the reformulation as a multi-agent coordination problem over an undirected graph. We exploited incremental passivity properties and showed that a synchronizing, distributed Laplacian feedback can be designed using relative estimates of the neighbours. Under strict monotonicity property of the pseudo-gradient, we proved that the new dynamics converges to the NE of the game. We discussed cases that highlight the tradeoff between properties of the game and the communication graph.