Dynamic NE Seeking for Multi-Integrator Networked Agents with Disturbance Rejection

TL;DR

This paper develops a disturbance-resilient Nash equilibrium seeking method for multi-integrator agents with partial information, using gradient-play and internal-model based observers, ensuring convergence despite external disturbances.

Contribution

It introduces a novel dynamic Nash equilibrium seeking approach combining gradient-play with disturbance observers for multi-integrator agents under partial information.

Findings

Convergence to Nash equilibrium is achieved despite disturbances.

The method applies to single, double, and multi-integrator agents.

The approach is proven using input-to-state stability and strong monotonicity.

Abstract

In this paper, we consider game problems played by (multi)-integrator agents, subject to external disturbances. We propose Nash equilibrium seeking dynamics based on gradient-play, augmented with a dynamic internal-model based component, which is a reduced-order observer of the disturbance. We consider single-, double- and extensions to multi-integrator agents, in a partial-information setting, where agents have only partial knowledge on the others' decisions over a network. The lack of global information is offset by each agent maintaining an estimate of the others' states, based on local communication with its neighbours. Each agent has an additional dynamic component that drives its estimates to the consensus subspace. In all cases, we show convergence to the Nash equilibrium irrespective of disturbances. Our proofs leverage input-to-state stability under strong monotonicity of the pseudo-gradient and Lipschitz continuity of the extended pseudo-gradient.

Equations

$$\dot{x}=f(x,u)$$

$$\displaystyle\|x(t)\|\leq\beta(\|x(t_{0})\|,t-t_{0})+\gamma\Big{(}\sup_{t_{o}\leq\tau\leq t}\|u(\tau)\|\Big{)},\,\,\forall t\geq t_{0}$$

$${\alpha }_1(\Vert x\Vert )$$

$$\frac{\partial V}{\partial x}f(x,u)$$

$${\dot{x}}_1$$

$${\dot{x}}_2$$

$$J_i(x_i^{\ast },x_{-i}^{\ast })\le J_i(x_i,x_{-i}^{\ast })\forall i\in \mathcal{I},\forall x_i\in {\Omega }_i$$

$${\nabla }_i{J}_i(x_i^{\ast },x_{-i}^{\ast })=0,\forall i\in \mathcal{I}$$

$$\displaystyle\Gamma_{NE}=\big{\{}x\in\mathbb{R}^{n}|\nabla_{i}J_{i}(x_{i},x_{-i})=0,\,\forall i\in\mathcal{I}$$

$$F(x^{\ast })=0$$

$${\Sigma }_i:{\dot{x}}_i=-{\nabla }_i{J}_i(x_i,x_{-i}),\forall i\in \mathcal{I}$$

$${\Sigma }_i:\{\begin{array}{l}{\dot{\text{x}}}_{-i}^i=-{\mathcal{S}}_i{\sum }_{j\in {\mathcal{N}}_i}({\text{x}}^i-{\text{x}}^j)\\ {\dot{x}}_i=-{\nabla }_i{J}_i(x_i,{\text{x}}_{-i}^i)-{\mathcal{R}}_i{\sum }_{j\in {\mathcal{N}}_i}({\text{x}}^i-{\text{x}}^j),\forall i\in \mathcal{I}\end{array}$$

$$\begin{array}{rl}{\mathcal{R}}_i& :=\left[\begin{array}{ccc}{0}_{n_i\times n_{<i}}& I_{n_i}& 0_{n_i\times n_{>i}}\end{array}\right]\\ {\mathcal{S}}_i& :=\left[\begin{array}{ccc}{I}_{n_{<i}}& 0_{n_{<i}\times n_i}& 0_{n_{<i}\times n_{>i}}\\ 0_{n_{>i}\times n_{<i}}& 0_{n_{>i}\times n_i}& I_{n_{>i}}\end{array}\right]\end{array}$$

$${\mathcal{R}}_i^T{\mathcal{R}}_i+{\mathcal{S}}_i^T{\mathcal{S}}_i=I_n,\forall i\in \mathcal{I}$$

$$\text{F}(\text{x})=\mathrm{col}({\nabla }_1{J}_1(x_1,{\text{x}}_{-1}^1),\dots ,{\nabla }_N{J}_N(x_N,{\text{x}}_{-N}^N)).$$

$$\text{F}({\text{1}}_N\otimes x^{\ast })=0$$

$$x_i^{(r_i)}=u_i+d_i,\forall i\in \mathcal{I}$$

$${\mathcal{D}}_i:$$

$${\dot{x}}_i=u_i+d_i,\forall i\in \mathcal{I}$$

$$\begin{array}{rl}{\dot{\text{x}}}_{-i}^i& =-{\mathcal{S}}_i\sum _{j\in {\mathcal{N}}_i}({\text{x}}^i-{\text{x}}^j)\\ {\dot{\xi }}_i& =S_i(K_i{x}_i+{\xi }_i)+K_i{\nabla }_i{J}_i(x_i,{\text{x}}_{-i}^i)\\ & +K_i{\mathcal{R}}_i\sum _{j\in {\mathcal{N}}_i}({\text{x}}^i-{\text{x}}^j)\\ u_i& =-{\nabla }_i{J}_i(x_i,{\text{x}}_{-i}^i)-{\mathcal{R}}_i\sum _{j\in {\mathcal{N}}_i}({\text{x}}^i-{\text{x}}^j)\\ & -D_i(K_i{x}_i+{\xi }_i),\forall i\in \mathcal{I}\end{array}$$

$${\Sigma }_i:\{\begin{array}{ll}{\dot{x}}_i& =-{\nabla }_i{J}_i(x_i,{\text{x}}_{-i}^i)-{\mathcal{R}}_i{\sum }_{j\in {\mathcal{N}}_i}({\text{x}}^i-{\text{x}}^j)\\ & -D_i(K_i{x}_i+{\xi }_i)+d_i\\ {\dot{\text{x}}}_{-i}^i& =-{\mathcal{S}}_i{\sum }_{j\in {\mathcal{N}}_i}({\text{x}}^i-{\text{x}}^j)\\ {\dot{\xi }}_i& =S_i(K_i{x}_i+{\xi }_i)+K_i{\nabla }_i{J}_i(x_i,{\text{x}}_{-i}^i)\\ & +K_i{\mathcal{R}}_i{\sum }_{j\in {\mathcal{N}}_i}({\text{x}}^i-{\text{x}}^j),\forall i\in \mathcal{I}\end{array}$$

$$\dot{w}=Sw$$

$$\Sigma :$$

$$\dot{w}$$

$$\begin{array}{rl}\dot{\text{x}}& =-{\mathcal{R}}^T\text{F}(\text{x})-\text{L}\text{x}+{\mathcal{R}}^{T}D\rho \\ \dot{\rho }& =(S-KD)\rho \end{array}$$

$$\begin{array}{rl}\dot{\tilde{\text{x}}}& =-{\mathcal{R}}^T\text{F}(\tilde{\text{x}}+\bar{\text{x}})-\text{L}(\tilde{\text{x}}+\bar{\text{x}})+{\mathcal{R}}^{T}D\rho \\ \dot{\rho }& =(S-KD)\rho \end{array}$$

$$\dot{V}$$

$$\begin{array}{rl}\dot{V}& =-({\tilde{\text{x}}}^{\perp }+{\tilde{\text{x}}}^{\Vert }{)}^T{\mathcal{R}}^T[\text{F}(\tilde{\text{x}}+\bar{\text{x}})-\text{F}(\bar{\text{x}})]\\ & -({\tilde{\text{x}}}^{\perp }+{\tilde{\text{x}}}^{\Vert }{)}^T\text{L}({\tilde{\text{x}}}^{\perp }+{\tilde{\text{x}}}^{\Vert })+({\tilde{\text{x}}}^{\perp }+{\tilde{\text{x}}}^{\Vert }{)}^T{\mathcal{R}}^{T}D\rho \end{array}$$

$$\begin{array}{rl}\dot{V}& \le -({\tilde{\text{x}}}^{\perp }{)}^T{\mathcal{R}}^T[\text{F}({\tilde{\text{x}}}^{\perp }+{\tilde{\text{x}}}^{\Vert }+\bar{\text{x}})-\text{F}({\tilde{\text{x}}}^{\Vert }+\bar{\text{x}})]\\ & -({\tilde{\text{x}}}^{\perp }{)}^T{\mathcal{R}}^T[F(x)-F(x^{\ast })]-{\lambda }_2(L)\Vert {\tilde{\text{x}}}^{\perp }{\Vert }^2\\ & -(x-x^{\ast }{)}^T[\text{F}({\tilde{\text{x}}}^{\perp }+{\tilde{\text{x}}}^{\Vert }+\bar{\text{x}})-\text{F}({\tilde{\text{x}}}^{\Vert }+\bar{\text{x}})]\\ & -(x-x^{\ast }{)}^T[F(x)-F(x^{\ast })]+({\tilde{\text{x}}}^{\perp }+{\tilde{\text{x}}}^{\Vert }{)}^T{\mathcal{R}}^{T}D\rho \end{array}$$

$$\begin{array}{rl}\dot{V}& \le \theta \Vert {\tilde{\text{x}}}^{\perp }{\Vert }^2+\theta \Vert {\tilde{\text{x}}}^{\perp }\Vert \Vert x-x^{\ast }\Vert -{\lambda }_2(L)\Vert {\tilde{\text{x}}}^{\perp }{\Vert }^2\\ & +\theta \Vert x-x^{\ast }\Vert \Vert {\tilde{\text{x}}}^{\perp }\Vert -\mu \Vert x-x^{\ast }{\Vert }^2+({\tilde{\text{x}}}^{\perp }+{\tilde{\text{x}}}^{\Vert }{)}^T{\mathcal{R}}^{T}D\rho \end{array}$$

$$\begin{array}{rl}\dot{V}& \le -\left[\begin{array}{cc}\Vert {\tilde{\text{x}}}^{\Vert }\Vert & \Vert {\tilde{\text{x}}}^{\perp }\Vert \end{array}\right]\left[\begin{array}{cc}\frac{1}{N}\mu & -\frac{1}{\sqrt{N}}\theta \\ -\frac{1}{\sqrt{N}}\theta & {\lambda }_2(L)-\theta \end{array}\right]\left[\begin{array}{c}\Vert {\tilde{\text{x}}}^{\Vert }\Vert \\ \Vert {\tilde{\text{x}}}^{\perp }\Vert \end{array}\right]\\ & +\Vert {\tilde{\text{x}}}^{\perp }+{\tilde{\text{x}}}^{\Vert }\Vert \Vert {\mathcal{R}}^{T}D\Vert \Vert \rho \Vert \end{array}$$

$$\begin{array}{rl}\dot{V}& \le -\left[\begin{array}{cc}\Vert {\tilde{\text{x}}}^{\Vert }\Vert & \Vert {\tilde{\text{x}}}^{\perp }\Vert \end{array}\right]\left[\begin{array}{cc}\frac{1}{N}\mu & -\frac{1}{\sqrt{N}}\theta \\ -\frac{1}{\sqrt{N}}\theta & {\lambda }_2(L)-\theta \end{array}\right]\left[\begin{array}{c}\Vert {\tilde{\text{x}}}^{\Vert }\Vert \\ \Vert {\tilde{\text{x}}}^{\perp }\Vert \end{array}\right]\\ & +a\Vert {\tilde{\text{x}}}^{\perp }+{\tilde{\text{x}}}^{\Vert }{\Vert }^2\end{array}$$

Full text

I Introduction

Game theory has found many applications in multi-agent engineering problems, wherein each agent can be modelled as an independent, selfish decision maker that tries to optimize its individual, but coupled, cost function. These include wireless communication networks [1], [2], [3], optical networks [4], [5], smart-grid and PEV charging [6], [7], [8] noncooperative flow control [9], [10] and multi-agent formation problems[11]. The relevant equilibrium sought is the Nash equilibrium (NE), whereby no agent has incentive to unilaterally change its action. The objective is to design either continuous-time or discrete-time, distributed learning schemes that converge to the NE under reasonable assumptions on the game properties and agent knowledge. Most works focus on algorithms for agents that either do not have dynamics, or have single integrator dynamics, and disturbances are not explicitly considered, [12].

There are many scenarios when the game or the agents are subject to disturbances, noise or uncertainties. Examples are demand-side management in smart-grids, with changes in the energy consumption demand, [6], feedback control for PEV charging load allocation, [8], or power control for optical-signal-to-noise ratio (OSNR) in the presence of pilot tones, [13]. Yet there have been relatively few works on Nash equilibrium seeking in such settings. In [8], a time-varying pricing function that affects the cost functions of each agent is considered. Only robustness to the time-varying component is investigated. Another good motivating example is the case of a group of mobile robots in a sensor network, similar to the examples in [14]. Each agent in the network has a goal related to its global position. However, it must also consider its position relative to the other agents in the network in order to make sure that it maintains communication with its neighbours. This can easily be formulated as a game played by the robots, which can be modelled as higher-order agents. In addition, each robot may be subject to a disturbance, e.g., wind or a slope in the terrain. It is important that these robots be able to reject this deterministic disturbance and still converge to the NE. A similar problem without disturbances was presented in [15], however the state space is discretized and the game is treated as a finite action game. This formulation ignores the dynamics of the individual agents.

Motivated by the above, in this paper our focus is to extend these results to games wherein the agents are modelled as (multi)-integrator systems subject to external deterministic disturbances. This is related to NE seeking with noisy feedback, on which there has been recent work. A dual-averaging algorithm with noisy gradients is considered in [16]. A discrete-time extremum seeking algorithm with noisy cost measurements for agents modelled as single and double integrators and kinematic unicycles is investigated in [14]. In both of these papers, the noise involved is stochastic in nature instead of a deterministic disturbance, as we consider here. Separately, NE seeking in the special class of aggregative games for Euler-Lagrange systems has been recently investigated in [17], which is similar to our work due to the dynamic nature of the agents involved, but without disturbances being considered.

Our work is related to the literature on disturbance rejection and tracking in multi-agent systems, [18], [19], [20], [21], [22]. Most output regulation problems in multi-agent systems can be viewed as specific cases of game theoretical problems. The synchronization problem, for example, can be regarded as a special game where each agent’s cost function is quadratic and corresponds to the sum of the squared distances to all of its neighbours.

Our work is also related to distributed optimization, where a group of agents cooperatively minimize a global cost function, the sum of the agents’ individual cost functions. Optimization schemes that reject disturbances have been discussed for single integrator systems [23], systems with unit relative degree [24] and systems with double integrator dynamics [25]. In [26] the robustness of a continuous-time distributed optimization algorithm is analyzed in the presence of additively persistent noise on agents’ communication and computation, in a directed communication graph. Key differences from a game setup are the cooperative nature of the problem and the fact that usually each agent’s cost is decoupled of the others’ variables. Exploiting summability, leads to a set of parallel decoupled optimization problems, one for each agent and its own cost function. Even when the overall cost is not separable, due to its summable structure, one can extend the problem to an augmented space of estimates, where it becomes separable and convex. In a game context, an agent’s cost is inherently coupled to the others’ decisions, on which it does not have control and convexity is only partial.

*Contributions. * Motivated by the above, in this paper we consider how to design Nash equilibrium seeking dynamics that simultaneously reject exogenous disturbances. We consider single and double-integrator agents, i.e., agents that behave as continuous-time dynamical systems that integrate their respective inputs, in a partial-information setting, i.e., networked regimes where agents may only access the states of their neighbours. We also discuss extensions to multi-integrator agents. Unlike multi-agent set stabilization problems with disturbance rejection, herein the stabilization goal is the a priori unknown Nash equilibrium of the game, which has to be reached irrespective of disturbances. In all cases, we make standard assumptions that provide existence and uniqueness of the NE of the game.

Due to the partial-information setting, we inspire from the disturbance-free results in [27]. Each player keeps track of an estimate of the others’ decisions as in [27], and the problem can be seen as one of multi-agent agreement with disturbance rejection. The agreement subspace is the estimate-consensus subspace at the Nash equilibrium, irrespective of the disturbance. The proposed agent learning dynamics has two components: a gradient-play with estimate consensus component (that drives each player’s dynamics towards minimizing its own cost function) and a dynamic internal-model component, which effectively implements a reduced-order observer of the disturbance. Unlike typical multi-agent agreement, [18], [19], [21], we cannot use individual passivity of each agent. Rather, our proofs rely on combining input-to-state stability with design of a reduced-observer for disturbance, under strong monotonicity of the pseudo-gradient and Lipschitz continuity of the extended pseudo-gradient. The resulting agent dynamics are locally distributed, with coupling introduced only through the communication graph.

The paper is organized as follows. In Section II, we give the necessary background on nonlinear systems, graph theory and noncooperative game theory. In Section III, we formulate the NE seeking problem for dynamic agents with disturbance rejection. In Section IV, we give our results for NE seeking dynamics with disturbance rejection for single-integrator agents. In Section V, we formulate a NE seeking algorithm for double-integrator agents and discuss extensions to multi-integrator agents. In Section VI, we compare by simulation their performance with those of a standard gradient-play dynamics and an augmented gradient-play dynamics with estimate consensus (partial information setting), and give conclusions in Section VII. A short version of this work appeared in [28], where only single-integrators are treated.

Notations. Let $\mathbb{R}$, $\mathbb{R}_{\geq 0}$ denote the set of real and non-negative real numbers, $\mathbb{C}$ and $\mathbb{C}^{-}$ the set of complex numbers and complex numbers with negative real part. Given $x,y\!\in\!\mathbb{R}^{n}$, $x^{T}y$ denotes the inner product of $x$ and $y$. Let $\|\cdot\|\!:\!\mathbb{R}^{n}\!\rightarrow\!\mathbb{R}_{\geq 0}$ denote the Euclidean norm and $\|\cdot\|\!:\!\mathbb{R}^{m\times n}\!\rightarrow\!\mathbb{R}_{\geq 0}$ denote the induced matrix norm. $\operatorname{col}(x_{1},\dots,x_{N})$ denotes $[x_{1}^{T},\dots,x_{N}^{T}]^{T}$. Given matrices $A_{1},\dots,A_{N}$, $\operatorname{blkdiag}(A_{1},\dots,A_{N})$ denotes the block diagonal matrix with $A_{i}$ on the diagonal. $I_{n}$ denotes the $n\!\times\!n$ identity matrix. $\boldsymbol{1}_{n}$ denotes the $n\!\times\!1$ all ones vector. $A\otimes B$ denotes the Kronecker product of matrices $A$ and $B$.

II Background

II-A Input to State Stability

In this work, we model the dynamics of each agent as a continuous time dynamical system. We first introduce some background from [29]. Consider a nonlinear system,

[TABLE]

where $\dot{x}:=\frac{dx(t)}{dt}$, $f\!:\!\mathbb{R}^{n}\!\times\!\mathbb{R}^{m}\!\rightarrow\!\mathbb{R}^{n}$ is locally Lipschitz in $x$ and $u$ and the input $u(t)$ is a piecewise continuous, bounded function.

Definition 1

System (1) is input-to-state stable (ISS) if there exist $\beta\in\mathcal{KL}$ and $\gamma\in\mathcal{K}$ such that for any initial state $x(t_{0})$ and any bounded input $u(t)$, the solution $x(t)$ satisfies

[TABLE]

Theorem 1

(Theorem 4.19, [29]) Let $V(x)$ be a continuously differentiable function such that

[TABLE]

$\forall x\!\in\!\mathbb{R}^{n},\ u\!\in\!\mathbb{R}^{m}$, where $\alpha_{1},\alpha_{2}\!\in\!\mathcal{K}_{\infty},\!\rho\!\in\!\mathcal{K}$, and $W(x)$ is positive definite. Then system (1) is ISS with $\gamma\!=\!\alpha_{1}^{-1}\!\circ\!\alpha_{2}\!\circ\!\rho$.

Consider now the cascade of two systems

[TABLE]

with $f_{1}\!:\!\mathbb{R}^{n_{1}}\!\times\!\mathbb{R}^{n_{2}}\!\rightarrow\!\mathbb{R}^{n_{1}}$, $f_{2}\!:\!\mathbb{R}^{n_{2}}\!\rightarrow\!\mathbb{R}^{n_{2}}$ locally Lipschitz.

Lemma 1

(Lemma 4.7, [29]) If the system (2) with $x_{2}$ as an input is ISS and the origin of (3) is globally uniformly asymptotically stable, then the origin of the cascade system (2) and (3) is globally uniformly asymptotically stable.

II-B Graph Theory

In this paper, we consider NE seeking for dynamic agents with communication over networks with fixed (static) topology. The communication protocol relies on graph theory. The following is from [30]. An undirected graph, $G$ is a pair $G=(\mathcal{I},E)$ where $\mathcal{I}=\{1,\dots,N\}$ is the vertex set and $E\subset\mathcal{I}\times\mathcal{I}$ is the edge set. Since $G$ is an undirected graph, for all $i,j\in\mathcal{I}$, if $(i,j)\in E$ then $(j,i)\in E$. Let $\mathcal{N}_{i}\subset\mathcal{I}$ denote the set of neighbours of player $i$. The adjacency matrix $\textbf{A}=[a_{ij}]\in\mathbb{R}^{N\times N}$ of the graph $G$ is defined such that $a_{ij}=1$ if $(j,i)\in E$ and $a_{ij}=0$ otherwise. For an an undirected graph, $a_{ij}=a_{ji}$. $G$ is connected if any two agents are connected by a path. The Laplacian matrix $L=[l_{ij}]\in\mathbb{R}^{N\times N}$ of the graph $G$ is defined as $l_{ii}=\sum_{j\neq i}a_{ij}=|\mathcal{N}_{i}|$ and $l_{ij}=-a_{ij}$, for $i\neq j$. For an undirected and connected graph, $L$ is symmetric positive definite and has a simple zero eigenvalue such that $0<\lambda_{2}(L)\leq\ldots\leq\lambda_{N}(L)$ and $L\textbf{1}_{N}=0$. Furthermore, for any vector $y\in\mathbb{R}^{N}$ satisfying $\textbf{1}_{N}^{T}y=0$, $\lambda_{2}(L)\|y\|^{2}\leq y^{T}Ly\leq\lambda_{N}(L)\|y\|^{2}$.

II-C Game Theory

Consider a set of players, $\mathcal{I}=\{1,\dots,N\}$. Each player $i\in\mathcal{I}$ controls its own action $x_{i}\in\Omega_{i}\subset\mathbb{R}^{n_{i}}$. The overall action set of the players is $\Omega=\Omega_{1}\times\dots\times\Omega_{N}\subset\mathbb{R}^{n}$, where $n=\sum_{i\in\mathcal{I}}n_{i}$. Let $x=(x_{i},x_{-i})\in\Omega$ denote the overall action profile of all players, where {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}x_{-i}}\in\Omega_{-i}=\Omega_{1}\times\dots\times\Omega_{i-1}\times\Omega_{i+1}\times\dots\times\Omega_{N}\subset\mathbb{R}^{n_{-i}} is the action set of all players except for player $i$. Let $J_{i}:\Omega\rightarrow\mathbb{R}$ be the cost function of player $i$. Each player tries to minimize its own cost function over its action. Denote the game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$.

Definition 2

Given a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$, an action profile $x^{*}=(x_{i}^{*},x_{-i}^{*})\in\Omega$ is a Nash Equilibrium (NE) of $\mathcal{G}$ if

[TABLE]

At a Nash Equilibrium no player can unilaterally decrease its cost, and thus has no incentive to switch strategies (actions) on its own.

Assumption 1

For each $i\in\mathcal{I}$, let $\Omega_{i}=\mathbb{R}^{n_{i}}$, the cost function $J_{i}:\Omega\rightarrow\mathbb{R}$ be $\mathcal{C}^{1}$ in its arguments and convex in $x_{i}$.

Under Assumption 1, any NE satisfies

[TABLE]

where $\nabla_{i}J_{i}(x_{i},x_{-i})=\frac{\partial}{\partial x_{i}}J_{i}(x_{i},x_{-i})\in\mathbb{R}^{n_{i}}$ is the partial gradient of player $i$’s cost function, with respect to its own action. We denote the set of all NE in the game by

[TABLE]

Let $\!F(x)\!\!=\!\!\operatorname{col}(\!\nabla_{1}J_{1}(x),\dots,\!\nabla_{N}J_{N}(x))$ denote the pseudo-gradient- the stacked vector of all partial gradients, so (4) is

[TABLE]

Assumption 2

The pseudo-gradient $F:\Omega\rightarrow\mathbb{R}^{n}$ is strongly monotone, $(x-x^{\prime})^{T}(F(x)-F(x^{\prime}))>\mu||x-x^{\prime}||^{2}$, $\forall x,x^{\prime}\in\mathbb{R}^{n}$ for $\mu>0$ and Lipschitz continuous, $||F(x)-F(x^{\prime})||\leq\theta||x-x^{\prime}||$, $\theta>0$.

Under Assumptions 1 and 2, by Theorem 3 in [31], the game has a unique NE.

II-D Full-Information Gradient Dynamics

In the rest of this paper, we assume that each agent updates its action in a continuous manner, therefore $x_{i}=x_{i}(t)$. For simplicity of notation, we drop the explicit dependence on time. In a game with perfect information, i.e., complete communication graph, a gradient-based NE seeking algorithm (gradient-play) can be used for action update, given by

[TABLE]

We call $\Sigma_{i}$ the agent learning dynamics, and note that it requires full decision information of the others’, $x_{-i}$.

The game can be visualized as an interconnection between all agents’ learning dynamics, $\Sigma_{i}$, $i\!\in\!\mathcal{I}$, represented as in Fig. 1, where $\Sigma_{-i}$ denote the other agents’ learning dynamics (except $i$), and $s_{-i}$ is the information received by agent $i$ from the others $\Sigma_{-i}$ in continuous-time. Hence, in the full information setting, $s_{-i}=x_{-i}$.

With (6), the overall dynamics of all players, $\Sigma=(\Sigma_{i},\Sigma_{-i})$ is $\Sigma:\,\,\,\dot{x}=-F(x)$. Note that $\Sigma$ can be viewed as a feedback interconnection between a bank of integrators with the pseudo-gradient map $F$. Under Assumption 1, the solutions of (6) exist and are unique for any initial condition, $x(0)$. Under Assumption 2, the unique Nash Equilibrium of the game is globally asymptotically stable for the interconnected $\Sigma$, with $\Sigma_{i}$ as in (6), (cf. [32] or Lemma 1, [27]).

II-E Partial-Information Gradient Dynamics

Often only partial information is available to each agent, i.e., from the neighbours of each agent. In this case, a modified algorithm must be used, where agent $i$ uses estimates, $\textbf{x}^{i}$, which it shares with its neighbours, and evaluates its gradient using these estimates instead of the others’ actions. Referring to Fig. 1, in this case, $s_{-i}=\{\textbf{x}^{j}|j\in\mathcal{N}_{i}\}$. The following is from [27]. Consider a game with information exchanged over a network, with static communication graph, $G_{c}$ and Laplacian, $L$.

Assumption 3

The undirected graph $G_{c}$ is connected.

Consider the following agent learning dynamics

[TABLE]

where $\textbf{x}_{-i}^{i}$ are agent $i$’s estimates of the others actions. Based on local communication with its neighbours, $\mathcal{N}_{i}$, each agent $i$ computes estimates of all other agents’ actions, $\textbf{x}_{-i}^{i}\!\!=\!\!\operatorname{col}(\textbf{x}_{1}^{i},\dots,\textbf{x}_{i-1}^{i},\textbf{x}_{i+1}^{i},\dots,\textbf{x}_{N}^{i})\!\in\!\mathbb{R}^{n_{-i}}$ and uses these estimates to evaluate its gradient, $\nabla_{i}J_{i}(x_{i},\textbf{x}_{-i}^{i})$. Then, $\textbf{x}^{i}\!=\!\operatorname{col}(\textbf{x}_{1}^{i},\dots,\textbf{x}_{i-1}^{i},x_{i},\textbf{x}_{i+1}^{i},\dots,\textbf{x}_{N}^{i})\!\in\!\mathbb{R}^{n}$ and $\textbf{x}\!=\!\operatorname{col}(\textbf{x}^{1},\dots,\textbf{x}^{N})\!\in\!\mathbb{R}^{Nn}$. The matrices $\mathcal{R}_{i}$ and $\mathcal{S}_{i}$ are defined as follows

[TABLE]

for actions and estimates selection, where $n_{<i}:=\sum_{j<i\ j,i\in\mathcal{I}}n_{j}$ and $n_{>i}:=\sum_{j>i\ j,i\in\mathcal{I}}n_{j}$. Then $x_{i}=\mathcal{R}_{i}\textbf{x}^{i}$, $\textbf{x}_{-i}^{i}=\mathcal{S}_{i}\textbf{x}^{i}$, and $\textbf{x}^{i}=\mathcal{R}_{i}^{T}x_{i}+\mathcal{S}_{i}^{T}\textbf{x}^{i}_{-i}$. Note that,

[TABLE]

The vector of stacked, partial gradients $\nabla_{i}J_{i}(x_{i},\textbf{x}_{-i}^{i})$ in (7), computed based on estimates, is denoted as

[TABLE]

and is called the extended pseudo-gradient. Note that F satisfies $\textbf{F}(\textbf{1}_{N}\otimes x)=F(x)$ for any $x$, hence

[TABLE]

Assumption 4

F* is Lipschitz continuous, $||\textbf{F}(\textbf{x})-\textbf{F}(\textbf{x}^{\prime})||\leq\theta||\textbf{x}-\textbf{x}^{\prime}||$, for all $\textbf{x},\textbf{x}^{\prime}\in\mathbb{R}^{Nn}$, for some $\theta>0$.*

Under Assumptions 1- 4, if $\mu(\lambda_{2}(L)-\theta)>\theta^{2}$, the unique NE, $x=x^{*}$, is globally asymptotically stable for all networked interconnected $\Sigma_{i}$, (7), (cf. Theorem 1, [27]).

III Problem Formulation

In this paper, we consider the problem of NE seeking for multi-integrator agents in the presence of additive disturbance signals. The dynamics of each agent can be modelled by the following linear system, of order $r_{i}\geq 1$

[TABLE]

where $x_{i}^{(r_{i})}:=\frac{d^{r_{i}}x_{i}(t)}{dt^{r_{i}}}$. Each agent has a cost function $J_{i}(x_{i},x_{-i})$ that it seeks to minimize. Agent $i$ is affected by disturbance, $d_{i}$, which can be modelled as being generated by

[TABLE]

where $w_{i}\in\mathbb{R}^{q_{i}}$, $d_{i}\in\mathbb{R}^{n_{i}}$. We assume that $\mathcal{W}_{i}$ a compact subset set of $\mathbb{R}^{q_{i}}$ and that $\mathcal{D}_{i}$ is marginally (neutrally) stable and observable, [33]. Let $\mathcal{W}=\mathcal{W}_{1}\times\dots\times\mathcal{W}_{N}$. This setting is motivated by cases where the agents in the game have inherent dynamics. This occurs, for example, in the case of a game played between a network of velocity-actuated (single-integrator) or force-actuated (double-integrator) robots whose costs depend upon their position only. The disturbances are the result of a deterministic effect from the physical nature of the systems, e.g., wind pushing the mobile robots. These disturbances have known form (e.g., constant) but unknown parameters (e.g., strength). Therefore, we assume that each agent knows its $S_{i}$ and $D_{i}$ only, but has no knowledge of the initial condition $w_{i}(0)\in\mathcal{W}_{i}$ or the resulting solution $w_{i}(t)$. These are standard assumptions in the output regulation literature. Now the problem becomes one of finding control inputs, $u_{i}$, that minimize the cost function $J_{i}(x_{i},x_{-i})$ while simultaneously rejecting the disturbances, i.e., designing dynamics $\Sigma_{i}$, under which the NE $x^{*}$ is asymptotically stable for the closed-loop irrespective of disturbances (Fig. 2). We consider separately the single-integrator agents (Section IV) and double-integrator agents (Section V), and indicate how to extend the results to multi-integrator agents.

In each case we consider a partial-decision information setting, under local knowledge and communication over a graph $G_{c}$. We will show that if each player uses a gradient-play dynamics combined with an internal-model correction term that implements a reduced order observer for $w_{i}$, [34], (and a consensus-based dynamics), then every solution of the stacked dynamics of all agents stays bounded and will converge to the NE, $x^{*}$, irrespective of disturbances $w\in\mathcal{W}$.

IV NE Seeking for Single-Integrator Agents

In this section, we consider a game $\mathcal{G}$ where each agent is modelled as

[TABLE]

where $d_{i}$ is generated by (13), as in the example of a network of velocity actuated robots, and has a cost $J_{i}(x_{i},x_{-i})$, which it seeks to minimize while rejecting disturbances. We consider that each agent has partial (networked) information from his neighbours over graph, $G_{c}$.

Under Assumptions 1 and 2, the game has a unique NE. Inspired by (7), our proposed $u_{i}$ is dynamic and is generated by

[TABLE]

where $K_{i}$ is chosen such that $\sigma(S_{i}-K_{i}D_{i})\subset\mathbb{C}^{-}$. Note that (15) has a gradient-play term (evaluated at estimates) as well as a dynamic component $\dot{\xi}_{i}$ to reject disturbances, combined with a dynamic Laplacian-based estimate-consensus component $\dot{\textbf{x}}_{-i}^{i}$, which in steady-state should bring all $\textbf{x}^{i}$ to the consensus subspace, $\textbf{x}^{i}=\textbf{x}^{j}$. This leads to $\Sigma_{i}$ given by,

[TABLE]

Compared to (7), (16) has an extra-component, $\xi_{i}$, that acts an internal model for the disturbance. The following result shows convergence to the NE irrespective of disturbances.

Theorem 2

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ with partial information over a graph $G_{c}$ with Laplacian $L$ and agent learning dynamics $\Sigma_{i}$, (16), where disturbance $d_{i}$ is as in (13). Under Assumptions 1, 2, 3 and 4, if $\mu(\lambda_{2}(L)-\theta)>\theta^{2}$, then $\bar{\textbf{x}}=\textbf{1}_{N}\otimes x^{*}$, where $x^{*}$ is the unique NE, is globally asymptotically stable for all networked interconnected $\Sigma_{i}$s, (16), for all $w\in\mathcal{W}$. Moreover, all players’ estimates converge globally to $\bar{\textbf{x}}=\textbf{1}_{N}\otimes x^{*}$, for all $w\in\mathcal{W}$.

Proof:

The idea of the proof is to express all agents’ interconnected dynamics as a closed-loop dynamical system for which the NE is shown to be globally asymptotically stable irrespective of disturbances. To show stability we use a suitable change of coordinates to put the system in cascade form. Then we exploit ISS properties induced by strong monotonicity of the pseudo-gradient and Lipschitz continuity of the extended pseudo-gradient.

In stacked form, using F, (10), (13), all interconnected $\Sigma_{i}$, (16), of all agents $i\in\mathcal{I}$, can be written as a closed-loop system

[TABLE]

with $\mathcal{R}\!\!=\!\!\operatorname{blkdiag}\!(\mathcal{R}_{1},\dots\mathcal{R}_{N})$, $\!\mathcal{S}\!=\!\operatorname{blkdiag}\!(\mathcal{S}_{1}\dots\!\mathcal{S}_{N})$, $\!\textbf{L}\!\!=\!L\otimes I_{N}$, $\!\textbf{x}\!=\operatorname{col}(\textbf{x}^{1},\dots\textbf{x}^{N})$, $\!\operatorname{col}(\textbf{x}_{-1}^{1},\dots,\textbf{x}_{-N}^{N}\!)\!=\!\mathcal{S}\textbf{x}\!$ by $\textbf{x}_{-i}^{i}\!=\!\mathcal{S}_{i}\textbf{x}^{i}$.

Consider the coordinate transformation $\xi\mapsto\rho\!:=\!w\!-\!(Kx\!+\!\xi)$, so that $\dot{\rho}\!=\!(S\!-\!KD)\rho$. Note that from $\textbf{x}^{i}=\mathcal{R}_{i}^{T}x_{i}+\mathcal{S}_{i}^{T}\textbf{x}^{i}_{-i}$ it follows that $\textbf{x}=\mathcal{R}^{T}x+\mathcal{S}^{T}\mathcal{S}\textbf{x}$. Using $\mathcal{R}^{T}\mathcal{R}\!+\!\mathcal{S}^{T}\mathcal{S}\!=\!I_{Nn}$, from (9), and the previous relations, it follows that in the new coordinates, the stacked-form dynamics (IV) are given as,

[TABLE]

We note that (18) is in cascade form from $\rho$ to x. By shifting the coordinates $\textbf{x}\mapsto\tilde{\textbf{x}}:=\textbf{x}-\bar{\textbf{x}}$, where $\bar{\textbf{x}}=\textbf{1}_{N}\otimes x^{*}$, the dynamics of the $(\tilde{\textbf{x}},\rho)$ subsystem become

[TABLE]

Note that (19) is again in cascade form, with the $\rho$-subsystem generating the external input for the $\tilde{\textbf{x}}$-subsystem. Consider $V(\tilde{\textbf{x}})=\frac{1}{2}\|\tilde{\textbf{x}}\|^{2}$. Then, along solutions of the $\tilde{\textbf{x}}$-subsystem in (19), using $\textbf{L}\!\bar{\textbf{x}}=\textbf{0}_{Nn}$, it holds that

[TABLE]

Decompose $\mathbb{R}^{Nn}$ as $\mathbb{R}^{Nn}\!=\!C^{n}\!\oplus\!E^{n}$, where $C^{n}\!=\!\{\textbf{1}_{N}\otimes\!x\!|\ x\!\in\!\mathbb{R}^{n}\}$ is the consensus subspace, and $E^{n}$ is its orthogonal complement. Any $\textbf{x}\!\in\!\mathbb{R}^{Nn}$ can be written as $\textbf{x \!}=\!\textbf{x}^{\bot}\!+\!\textbf{x}^{\|}$, where $\textbf{x}^{\|}\!=\!P_{C}\textbf{x}\!\in\!C^{n}$, $\textbf{x}^{\bot}\!=\!P_{E}\textbf{x}\!\in\!E^{n}$, for $P_{C}\!=\!\frac{1}{N}\textbf{1}_{N}\!\otimes\!\textbf{1}_{N}^{T}\!\otimes\!I_{n}$, $P_{E}\!=\!I_{Nn}\!-\!\frac{1}{N}\textbf{1}_{N}\!\otimes\!\textbf{1}_{N}^{T}\!\otimes\!I_{n}$. Thus, ${\textbf{x}}^{\|}\!=\!\textbf{1}_{N}\!\otimes\!x$, for some $x\!\in\!\mathbb{R}^{n}$, and $\tilde{\textbf{x}}\!=\!\textbf{x}\!-\!\bar{\textbf{x}}\!=\!\tilde{\textbf{x}}^{\bot}\!+\!\tilde{\textbf{x}}^{\|}$, where $\tilde{\textbf{x}}^{\|}\!=\!\textbf{1}_{N}\!\otimes\!(x\!-\!x^{*})$, $\tilde{\textbf{x}}^{\bot}\!=\!{\textbf{x}}^{\bot}$. Using $\textbf{F}(\bar{\textbf{x}})\!=\!\textbf{0}_{n}$ by (11), from (20) we get

[TABLE]

Note that $\textbf{L}\tilde{\textbf{x}}^{\|}\!=\!\textbf{0}_{Nn}$ and $\lambda_{2}(L)\|\tilde{\textbf{x}}^{\bot}\|^{2}\!\leq\!\tilde{\textbf{x}}^{\bot}\textbf{L}\tilde{\textbf{x}}^{\bot}$, $\forall\tilde{\textbf{x}}^{\bot}\!\in\!E^{n}$ by properties of the Laplacian under Assumption 3. Adding and subtracting $\!\textbf{F}(\tilde{\textbf{x}}^{\|}\!+\!\bar{\textbf{x}})$ in (21), with $\textbf{F}(\tilde{\textbf{x}}^{\|}\!+\!\bar{\textbf{x}})\!=\!\textbf{F}(\textbf{1}_{N}\otimes x)\!=\!F(x)$, $\textbf{F}(\bar{\textbf{x}})\!=\!\textbf{F}(\textbf{1}_{N}\otimes x^{*})\!=\!F(x^{*})$, and using $\mathcal{R}(\tilde{\textbf{x}}^{\|})\!=\!x-x^{*}$, yields

[TABLE]

Using $\|\textbf{F}(\!\tilde{\textbf{x}}^{\bot}\!+\!\tilde{\textbf{x}}^{\|}\!+\!\bar{\textbf{x}})\!-\!\textbf{F}(\!\tilde{\textbf{x}}^{\|}\!+\!\bar{\textbf{x}})\|\!\leq\!\theta\|\!\tilde{\textbf{x}}^{\bot}\|$ by Assumption 4, $\|\mathcal{R}\!\tilde{\textbf{x}}^{\bot}\|\!\leq\!\|\mathcal{R}\|\|\tilde{\textbf{x}}^{\bot}\|$, $\|F(x)-F(x^{*})\|\!\leq\!\bar{\theta}\|x-x^{*}\|\!\leq\!\theta\|x-x^{*}\|$, $(x-x^{*})^{T}[F(x)-F(x^{*})]\!\geq\!\mu\|x-x^{*}\|^{2}$ by Assumption 2, yields

[TABLE]

Using $\|x-x^{*}\|=\frac{1}{\sqrt{N}}\|\tilde{\textbf{x}}^{\|}\|$, we can write

[TABLE]

Then, given any $a>0$, for any $\|\tilde{\textbf{x}}^{\bot}+\tilde{\textbf{x}}^{\|}\|\geq\frac{\|\mathcal{R}^{T}D\|}{a}\|\rho\|$,

[TABLE]

Note that $\|\tilde{\textbf{x}}^{\bot}+\tilde{\textbf{x}}^{\|}\|^{2}=\|\tilde{\textbf{x}}^{\bot}\|^{2}+\|\tilde{\textbf{x}}^{\|}\|^{2}=\|\tilde{\textbf{x}}\|^{2}$, so that, for any $\|\tilde{\textbf{x}}^{\bot}+\tilde{\textbf{x}}^{\|}\|\geq\frac{\|\mathcal{R}^{T}D\|}{a}\|\rho\|$ we can write,

[TABLE]

For the $\tilde{\textbf{x}}$-subsystem in (19) to be ISS, we need the matrix on the right-hand side to be positive definite. This holds for any $a\!>\!0$ such that $a\!<\!\frac{1}{N}\mu$, $a\!<\!\lambda_{2}(L)\!-\!\theta$, and $(\frac{1}{N}\mu\!-\!a)(\lambda_{2}(L)\!-\!\theta\!-\!a)\!-\!\frac{1}{N}\theta^{2}\!>\!0$. Since $\mu(\lambda_{2}(L)-\theta)>\theta^{2}$, the intersection of the above inequalities is guaranteed to be nonempty and the matrix is positive definite for any such $a$. Then, for any such $a$, $\dot{V}(\tilde{\textbf{x}})\leq-W(\tilde{\textbf{x}}),\,\forall\|\tilde{\textbf{x}}\|\geq\frac{\|\mathcal{R}^{T}D\|}{a}\|\rho\|$, where $W(\tilde{\textbf{x}})$ is a positive definite function, hence the $\tilde{\textbf{x}}$-subsystem in (19) is ISS with respect to $\rho$ by Theorem 1. Since $\!\dot{\rho}\!=\!(S\!-\!KD)\rho$ is asymptotically stable by (16), it follows that the origin of (19) is asymptotically stable by Lemma 1, hence $(\mathbf{1}_{N}\!\otimes x^{*},0)$ is asymptotically stable for (18), for any $w\in\mathcal{W}$. ∎

Remark 1

Local results follow if Assumption 4 holds only locally around $\mathbf{x}^{*}=\mathbf{1}_{N}\otimes x^{*}$. We note that the class of quadratic games satisfies Assumption 4 globally.

Remark 2

In the special case of full-information, there is no need for estimates and the agent (closed-loop) learning dynamics $\Sigma_{i}$, (16), reduce to,

[TABLE]

*The convergence result as in Theorem 2 holds without the need for Assumptions 3 and 4. *

V NE Seeking For Double Integrators

In this section, we consider NE seeking for double-integrator agents with disturbances. Our motivation is two-fold. Firstly, the agents in the game might have some sort of inherent dynamics, such as double integrator robots playing a game wherein the cost functions are functions of their positions. Each agent, therefore, cannot directly update its action, $x_{i}$, via choice of input $u_{i}$ and must take into account its inherent dynamics. Secondly, we may want to consider higher-order dynamics for learning, as done extensively in the optimization literature, e.g., the heavy-ball method.

Consider that each agent is modelled as a double integrator

[TABLE]

where $x_{i},v_{i},u_{i},d_{i}\in\mathbb{R}^{n_{i}}$, $d_{i}$ generated by (13). Each agent minimizes its cost function $J_{i}(x_{i},x_{-i})$, with the constraint that its steady-state velocity is zero. This setting is motivated for example in the case of a network of mobile, force-actuated robots whose costs depend on their positions only. At steady state, necessarily, their velocities must be zero. This requirement can be seen as the result of a quadratic penalty term, $J_{v_{i}}(v_{i})\!=\!\frac{1}{2}\|v_{i}\|^{2}$, on the velocity of each agent. Thus, the overall cost function for each agent is given by $\bar{J}(x_{i},x_{-i},v_{i})\!=\!J(x_{i},x_{-i})\!+\!\frac{1}{2}\|v_{i}\|^{2}$ and the resulting NE is

[TABLE]

Under Assumptions 1 and 2, $x^{*}$ is unique. By (25), $(x^{*},v^{*})$ is such that

[TABLE]

We consider partial-information learning dynamics under which the NE of the game is reached in the presence of additive disturbances.

We propose the following dynamic feedback

[TABLE]

where $K_{i}$ is such that $\sigma(S_{i}\!-\!K_{i}D_{i})\!\!\subset\!\mathbb{C}^{-}$ and $b_{i}\!>\!0$ and $\boldsymbol{\gamma}^{i}$ is agent $i$’s estimate variable. Note that in $\nabla_{i}J_{i}$ is evaluated at a predicted point, $x_{i}+b_{i}v_{i}$. We denote $\boldsymbol{\gamma}_{j}^{i}$ agent $i$’s prediction of $x_{j}+b_{j}v_{j}$. (27) uses a similar internal model and Laplacian-based consensus scheme (15), as in Section IV. However, instead of each agent estimating the others’ actions, $x_{-i}$, they estimate the predicted actions $\{x_{j}+b_{j}v_{j}|j\in\mathcal{I},j\neq i\}$ used to evaluate the gradient at the predicted point. Each agent will then share these estimates as well as their own prediction with their neighbours. Therefore, each agent $i$ computes $\boldsymbol{\gamma}_{-i}^{i}=\operatorname{col}(\boldsymbol{\gamma}_{1}^{i},\dots,\boldsymbol{\gamma}_{i-1}^{i},\boldsymbol{\gamma}_{i+1}^{i},\dots,\boldsymbol{\gamma}_{N}^{i})\in\mathbb{R}^{n_{-i}}$ and uses these estimates when evaluating its gradient, $\nabla_{i}J_{i}(x_{i}+b_{i}v_{i},\boldsymbol{\gamma}_{-i}^{i})$. Intuitively, each agent makes a prediction on the future state of the game, $\!x_{i}\!+\!b_{i}v_{i}$, based on the current actions and velocities, and evaluates its gradient with respect to $x_{i}$ at this point. We denote $\boldsymbol{\gamma}^{i}=\operatorname{col}(\boldsymbol{\gamma}_{1}^{i},\dots,\boldsymbol{\gamma}_{i-1}^{i},x_{i}+b_{i}v_{i},\boldsymbol{\gamma}_{i+1}^{i},\dots,\boldsymbol{\gamma}_{N}^{i})\in\mathbb{R}^{n}$ and $\boldsymbol{\gamma}=\operatorname{col}(\boldsymbol{\gamma}^{1},\dots,\boldsymbol{\gamma}^{N})\in\mathbb{R}^{Nn}$.

Remark 3

From an agent perspective, the intuition behind (27) is that each agent evaluates its partial-gradient at a predicted future point, $x_{i}+b_{i}v_{i}$, obtained as a first-order prediction from the current action and velocity of each agent, with a negative feedback on its velocity. This can be viewed as resulting from the quadratic penalty term associated with the velocity of each agent. In addition, consider the disturbance-free case and recall that gradient-play is a method that works well for single-integrators, i.e., systems with unit relative degree. By creating a fictitious output $\boldsymbol{\gamma}^{i}_{i}\!:\!=\!x_{i}\!+\!b_{i}v_{i}$ we decrease the relative degree of each agent to $\{1,\dots,1\}$. This creates a hyperplane $x+\mathcal{B}v-x^{*}=0$, where $\mathcal{B}=\operatorname{blkdiag}(b_{1}I_{n_{1}},\dots,b_{N}I_{n_{N}})$, on which the pseudo-gradient map is zero. The pseudo-gradient feedback makes this hyperplane attractive for the double-integrator system. The feedback stabilizes $v=0$ and renders this hyperplane invariant, thereby stabilizing $x=x^{*}$.

Remark 4

We note that (27) is similar to a passivity-based group coordination design, e.g., [35]. Indeed, the inner-loop feedback $u_{i}=-\frac{1}{b_{i}}v_{i}$ renders the agent dynamics passive with $\boldsymbol{\gamma}^{i}_{i}=x_{i}+b_{i}v_{i}$ as output. However, we stress that the feedback $\nabla_{i}J_{i}(x_{i}+b_{i}v_{i},\boldsymbol{\gamma}_{-i})$ is not necessarily the proper gradient of any function, as required in [35]. Therefore, individually, each agent is not a passive system when the feedback is added, due to coupling to the others’ actions via the cost function. This precludes using a passivity approach as in [35]. Rather, here we use a combined ISS approach to deal with both the disturbance and the higher-order stabilization.

The choice of feedback (27) yields learning (closed-loop) dynamics given by

[TABLE]

Theorem 3

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\mathbb{R}^{n_{i}})$ with partial information over a graph $G_{c}$ with Laplacian $L$ and learning dynamics $\Sigma_{i}$, (28), where disturbance $d_{i}$ is generated by (13). Under Assumptions 1-4, if $\mu(\lambda_{2}(L)\!-\!\theta)\!>\!\theta^{2}$ then $1_{N}\otimes x^{*}$, where $x^{*}$ is the unique NE, is globally asymptotically stable for all networked interconnected $\Sigma_{i}$, for all $w\in\mathcal{W}$. Moreover, each player’s estimates converge globally to the NE value, $\bar{\boldsymbol{\gamma}}=\textbf{1}_{N}\otimes x^{*}$.

Proof:

The idea of the proof is similar to that of Theorem 2. We use a change of coordinates to express the closed-loop dynamics in cascade form and use ISS arguments the show stability of the NE for the overall cascade system, irrespective of disturbance. The difference lies in the fact that (28) has extra terms due to the higher-order dynamics $\dot{v}_{i}$ that must be incorporated into the cascade.

The stacked dynamics of (28) is given by

[TABLE]

Note that $\mathcal{R}\boldsymbol{\gamma}=[\mathcal{R}_{i}\boldsymbol{\gamma}^{i}]_{i\in\mathcal{I}}=[x_{i}+b_{i}v_{i}]_{i\in\mathcal{I}}=x+\mathcal{B}v$. Let the coordinate transformation $x\mapsto\mathcal{R}\boldsymbol{\gamma}\!:=\!x\!+\!\mathcal{B}v$. Then,

[TABLE]

Combining this with the second equation in (29), by using the properties of $\mathcal{R}$ and $\mathcal{S}$, $\mathcal{R}^{T}\mathcal{R}+\mathcal{S}^{T}\mathcal{S}=I$, yields that

[TABLE]

Let $\xi\mapsto\rho\!:=\!w-(Kv\!+\!\xi)$, so that $\dot{\rho}=(S-KD)\rho$. Consider also $\boldsymbol{\gamma}\mapsto\tilde{\boldsymbol{\gamma}}:=\boldsymbol{\gamma}-\bar{\boldsymbol{\gamma}}$. Then, in the new coordinates, using $\textbf{L}\bar{\boldsymbol{\gamma}}=0$, the dynamics of the $(\tilde{\boldsymbol{\gamma}},v,\rho)$ are given by

[TABLE]

Note that the (30-32) is in cascade form with subsystem $(\tilde{\boldsymbol{\gamma}},\rho)$ (31, 32) generating the external input for (30). In turn, (31, 32) is in cascade form with (32) generating input $\rho$ for (31). We show first that (31) is ISS with respect to $\rho$. Consider

[TABLE]

which is positive definite. Taking the time-derivative of $V$ (33) along the solutions of (31), using $\textbf{F}(\bar{\boldsymbol{\gamma}})=0$, cf. (11), and properties of $\mathcal{R}$, $\mathcal{S}$, e.g. $\mathcal{R}\mathcal{S}^{T}=0$, $\mathcal{R}\mathcal{R}^{T}=I$, yields

[TABLE]

which is similar to (20). Then, following an argument as in the proof of Theorem 2, it follows that the $\tilde{\boldsymbol{\gamma}}$ subsystem of (31) is ISS with input $\rho$. Since the origin of the $\rho$ subsystem is globally asymptotically stable, then the origin of the $(\tilde{\boldsymbol{\gamma}},\rho)$ subsystem is globally asymptotically stable (cf. Lemma 1).

Now consider the $v$-subsystem (30) with input $(\tilde{\boldsymbol{\gamma}},\rho)$ and $V_{2}(v)=\frac{1}{2}\|v\|^{2}$. Along (30), using Assumption 2,

[TABLE]

where $b_{m}\!=\!\max_{i\in\mathcal{I}}b_{i}$, $\beta\!=\!\max\{\|\mathcal{R}\textbf{L}\|\!+\!\theta,\|D\|\}$. Thus $\dot{V}_{2}\leq-\frac{1}{b_{m}}\|v\|^{2}+\beta\|v\|\sqrt{2(\|\rho\|^{2}+\|\tilde{\boldsymbol{\gamma}}\|^{2})}$ or,

[TABLE]

where $\tilde{u}:=\operatorname{col}(\tilde{\boldsymbol{\gamma}},\rho)$. Hence, \dot{V}_{2}\!\leq\!-\big{(}\frac{1}{b_{m}}\!-\!b\big{)}\|v\|^{2},\ \forall\|v\|\!\geq\!\frac{\sqrt{2}\beta}{b}\!\|\tilde{u}\|, for any $0\!<\!b\!<\!\frac{1}{b_{m}}$. Therefore, by Theorem 1, (30) is ISS with $\tilde{u}=(\tilde{\boldsymbol{\gamma}},\rho)$. Since the origin of $(\tilde{\boldsymbol{\gamma}},\rho)$ subsystem is globally asymptotically stable, by Lemma 1, the origin of (31)-(32) is globally asymptotically stable, hence $(x^{*},0)$ is globally asymptotically stable for (29) for all $w\in\mathcal{W}$. ∎

Remark 5

In the full information case, there is no need for estimate and (28) reduces to

[TABLE]

The convergence results hold without the need for Assumptions 3 and 4. Furthermore, the disturbance-free, higher-order learning dynamics generated by (34) is

[TABLE]

which resembles heavy-ball with friction dynamics used in optimization, [36, 37].

Remark 6

The results from this section can easily be extended to multi-integrator agents. Consider that each agent is modelled as a $\!r_{i}^{th}\!$ order integrator, $r_{i}\!\geq\!2$,

[TABLE]

where $A_{i}\!=\!\begin{bmatrix}\!0_{n_{i}(r_{i}-2)\!\times\!n_{i}}\!&\!I_{n_{i}(r_{i}-2)}\\ \!0_{n_{i}\!\times\!n_{i}}\!&\!\!0_{n_{i}\!\times\!n_{i}(r_{i}-2)}\end{bmatrix}$, $B_{i}\!=\!\begin{bmatrix}\!0_{n_{i}(r_{i}-2)\!\times\!n_{i}}\\ \!I_{n_{i}}\!\end{bmatrix}$, $C_{i}\!=\!\begin{bmatrix}\!I_{n_{i}}\!\!&\!0_{n_{i}\!\times\!n_{i}(r_{i}-2)}\!\end{bmatrix}$, $v_{i}=\operatorname{col}(v_{i}^{1},\dots,v_{i}^{r_{i}-1})$, and has a cost function $J_{i}(x_{i},x_{-i})$. In this case, $\gamma_{i}\!:=\!x_{i}\!+\!\begin{bmatrix}c_{i}^{T}\!\otimes\!I_{n_{i}}\!&\!I_{n_{i}}\end{bmatrix}\!v_{i}$ where $c_{i}^{T}\!=\!\begin{bmatrix}c_{i,1}\!&\!\dots\!&\!c_{i,(r_{i}-2)}\end{bmatrix}$, $c_{i,k}$ are the coefficients of any $\!(r_{i}\!-\!1)^{\!th}\!$ order Hurwitz polynomial with $c_{i,0}\!\!=\!1$, $c_{i,(r_{i}-1)}\!\!=\!1$, and $u_{i}\!:=\!-\nabla_{i}J_{i}(\gamma_{i},\boldsymbol{\gamma}^{i}_{-i})\!-\!\begin{bmatrix}I_{n_{i}}\!&\!c_{i}^{T}\!\otimes\!I_{n_{i}}\end{bmatrix}\!v_{i}$. When $r_{i}\!=\!2$, this feedback reduces to the one for the second-order integrator with $b_{i}=1$. Then a dynamic learning scheme similar to (28) can be developed, by appropriately augmenting with reduced-order observer for the disturbance, and consensus-dynamics for the estimates $\boldsymbol{\gamma}^{i}_{-i}$.

The resulting agent learning dynamics are given as

[TABLE]

which for $r_{i}=2$ reduces to (28).

Theorem 4

Consider a game $\mathcal{G}(\mathcal{I},J_{i},\Omega_{i})$ with partial information communicated over a graph $G_{c}$ with Laplacian $L$ and agent dynamics given by $\Sigma_{i}$, (35). Under Assumptions 1, 2, 3 and 4, if $\mu(\lambda_{2}(L)-\theta)>\theta^{2}$ then the unique NE, $x=x^{*}$, is globally asymptotically stable for (35) for all $w\in\mathcal{W}$. Moreover, each player’s estimates converge globally to the NE values, $\bar{\textbf{x}}=\textbf{1}_{N}\otimes x^{*}$, for all $w\in\mathcal{W}$.

Proof:

Similar to Theorem 3. ∎

VI Numerical Examples

In this section we consider two application scenarios: an optical network OSNR game and a sensor network game. In both examples, our algorithms are compared with the full and partial-information gradient-play in the presence of disturbances.

VI-A OSNR Game

Consider an optical signal-to-noise ratio (OSNR) model for wavelength-division multiplexing (WDM) links [4], where 10 channels, $\mathcal{I}=\{1,\dots,10\}$, are transmitted over an optically amplified link. We consider each channel as an agent and denote each agent’s transmitting power as $x_{i}$, while the noise power of each channel as $n_{i}^{0}$. Each agent attempts to maximize its OSNR on its channel by adjusting its transmission power. Each agent has a cost function as in [38], given by

[TABLE]

where $a_{i}>0$ is a pricing parameter, $P^{0}$ is the total power target of the link, $b_{i}>0$, and $\Gamma=[\Gamma_{ij}]$ is the link system matrix, with parameters as in [39]. Each channel (agent) has dynamics (14), where the disturbance is generated due to the pilot-tones used for network tracing and monitoring, [13], which take the form of a sinusoidal signal with a unique frequency assigned for each channel and unknown modulation. Thus $d_{i}=P^{0}[1+m_{i}\sin(2\pi f_{i}t)]$, where $m_{i}=0.1i$ (unknown modulation index) and frequency $f_{i}=10i$ kHz, $i\in\mathcal{I}$.

First, we consider that each agent has full information about the others’ actions and we compare the results of agent dynamics, (23), with a standard gradient-play scheme (6). As seen in Fig. 3 and Fig. 4, (6) do not the reject disturbances (sustained fluctuations in the OSNR values), while (23) successfully reject disturbances and converge to the NE found in [39]. Next, assume each agent has partial information over a random graph, $G_{c}$, Fig. 5. The results of dynamics (16) are plotted in Fig. 7, while those of the Laplacian-based gradient dynamics (7) are shown in Fig. 6, with similar comparison.

VI-B Sensor Networks

Our next example is similar to the one investigated in [14]. However, our algorithm uses a continuous-time gradient-play inspired feedback instead of the discrete-time extremum seeking algorithm used in [14]. It is also important to note that while [14] considers noisy feedbacks, it does not consider disturbance rejection as we have posed it here.

Consider a group of five mobile robots in the plane in a sensor network. Each agent has a cost function that is a function of all robots’ positions, $(x_{i},x_{-i})$,

[TABLE]

where $r_{1}\!=\!\operatorname{col}(2,-2)$, $r_{2}\!=\!\operatorname{col}(-2,-2)$, $r_{3}\!=\!\operatorname{col}(-4,2)$, $r_{4}\!=\!\operatorname{col}(2,-4)$, and $r_{5}\!=\!\operatorname{col}(3,3)$. We consider two types, velocity actuated and force-actuated robots, and in each case we consider the full-information and the partial-information case with communication over a random graph, $G_{c}$ (Fig. LABEL:fig:Gc11).

VI-B1 Velocity-Actuated Robots

Consider that each agent in the network is a velocity-actuated robot with dynamics given by (14), where $d_{i}=\operatorname{col}(0.5,0)$ is a constant disturbance.

We consider first that each agent has full information about the other’s actions and compare our algorithm (23) to the standard gradient-play (6). In Fig. 9 solid-lines depict gradient-play results in the disturbance-free case. In the presence of disturbances, as seen in Fig. 9, (23) (dashed-lines) converges to the same NE values, while the standard gradient-play (dotted-lines) does not.

Next consider that each agent only has partial-information communicated over a graph, $G_{c}$. We compare our algorithm (16) to that of the Laplacian based-gradient dynamics (7) in Fig. 10, where solid-lines depict (7) results in the disturbance-free case. In the presence of disturbances, Fig. 10 shows that (16) (dashed-lines) converges to the same NE values as found by the full-information case, Fig. 9, while (7) (dotted-lines) does not.

VI-B2 Force-Actuated Robots

Consider that each agent is modelled as double integrator, (24) where $d_{i}=\operatorname{col}(0.5,0)$. The corresponding results are shown in Fig. 11 (full information case) and Fig. 12 (partial-information over $G_{c}$), where dashed-lines correspond to (34) and (28), respectively, while dotted-lines to the disturbance-free learning algorithm.

Remark 7

Although we did not specifically investigate systems with noisy feedbacks, it is possible to show that due to their ISS properties, the dynamics (16) and (28) have a certain amount of robustness to feedback noise, such as the type investigated in [14] and [16]. The ISS property implies that for any bounded feedback noise, the steady-state solution will remain in a neighbourhood of the NE.

VII Conclusions

We considered Nash equilibrium seeking schemes for (multi)-integrator agents subject to external disturbances. We addressed the case of full information on the others’ decisions, as well as the case where agents have partial-decision information, based on local observation and communication. In both cases, we proposed new continuous-time dynamic schemes that converge to the Nash equilibrium, irrespective of the disturbance. Besides a gradient-play component, the proposed agent dynamics have a dynamic internal-model component, and, in the case of partial-information, a consensus component that drives agents to reach the decision-estimate consensus subspace.