Stochastic Stability Analysis of Perturbed Learning Automata with Constant Step-Size in Strategic-Form Games

TL;DR

This paper analyzes the stochastic stability of perturbed learning automata with constant step-size in strategic-form games, extending convergence results beyond potential games by characterizing invariant measures of the induced Markov chain.

Contribution

It introduces a new framework for stochastic-stability analysis that does not rely on Lyapunov functions, enabling broader applicability to various game types.

Findings

Convergence to pure Nash equilibria demonstrated in potential games.

New stability analysis framework based on invariant probability measures.

Broader applicability beyond potential games achieved.

Abstract

This paper considers a class of reinforcement-learning that belongs to the family of Learning Automata and provides a stochastic-stability analysis in strategic-form games. For this class of dynamics, convergence to pure Nash equilibria has been demonstrated only for the fine class of potential games. Prior work primarily provides convergence properties of the dynamics through stochastic approximations, where the asymptotic behavior can be associated with the limit points of an ordinary-differential equation (ODE). However, analyzing global convergence through the ODE-approximation requires the existence of a Lyapunov or a potential function, which naturally restricts the applicabity of these algorithms to a fine class of games. To overcome these limitations, this paper introduces an alternative framework for analyzing stochastic-stability that is based upon an explicit characterization of the (unique) invariant probability measure of the induced Markov chain.

Equations

$${\mathcal{N}}_{\delta }(x)\doteq \{y\in \mathcal{X}:|x-y|<\delta \},$$

$$\Delta (n)\doteq \left\{x\in {\mathbb{R}}^n:x\ge 0,1^{\mathrm{T}}x=1\right\}.$$

$${\mathbb{I}}_A(x)\doteq \{\begin{array}{ll}1& \text{mbox}ifx\in A,\\ 0& \text{mbox}else.\end{array}$$

$${\alpha }_i(t+1)=\{\begin{array}{ll}{\mathrm{rand}}_{x_i(t)}[{\mathcal{A}}_i],& \text{mbox}withprobability1-\lambda ,\\ {\mathrm{rnad}}_{\mathrm{unif}}[{\mathcal{A}}_i],& \text{mbox}withprobability\lambda ,\end{array}$$

$$x_i(t+1)$$

$$=$$

$$\doteq$$

$$P_{\lambda }f(z)\doteq {\int }_{\mathcal{Z}}{P}_{\lambda }(z,dy)f(y),$$

$$\mu [f]\doteq {\int }_{\mathcal{Z}}\mu (dx)f(z),\text{mbox}for\mu \in \mathfrak{P}(\mathcal{Z}).$$

$$P(Z^{(k)}=({\alpha }^{(k)},x^{(k)}),O)$$

$$P(Z=(\alpha ,x),O)$$

$$\underset{k\to \infty }{\lim }P(Z^{(k)},O)\ge P(Z,O)=0.$$

$$({\mu }_{\lambda }{P}_{\lambda })(A)\doteq {\int }_{\mathcal{Z}}{\mu }_{\lambda }(dx)P_{\lambda }(z,A)={\mu }_{\lambda }(A),A\in \mathfrak{B}(\mathcal{Z}).$$

$${\hat{P}}_{s{s}^{\prime }}\doteq \underset{t\to \infty }{\lim }Q{P}^t(s,{\mathcal{N}}_{\delta }(s^{\prime })),$$

$$A_t$$

$$B_t$$

$$A_{\infty }\doteq \bigcup _{t=0}^{\infty }{A}_t\text{mbox}and{B}_{\infty }\doteq \bigcap _{t=1}^{\infty }{B}_t.$$

$$x_i(t)=e_{{\alpha }_i}-(1-ϵu_i(\alpha ){)}^t(e_{{\alpha }_i}-x_i(0)).$$

$${\mathbb{P}}_z[B_{\infty }]=\underset{t\to \infty }{\lim }{\mathbb{P}}_z[B_t]=\underset{t\to \infty }{\lim }\prod _{k=0}^t\prod _{i=1}^n{x}_{i{\alpha }_i}(k).$$

$$\sum _{t=1}^{\infty }\log (x_{i{\alpha }_i}(t))>-\infty .$$

$$y_i(t)\doteq 1-x_{i{\alpha }_i}(t)=\sum _{j\in {\mathcal{A}}_i\backslash {\alpha }_i}{x}_{ij}(t),$$

$$-\sum _{t=0}^{\infty }\log (1-y_i(t))<\infty ,\text{mbox}foralli\in \mathcal{I}.$$

$$\underset{t\to \infty }{\lim }\frac{-\log (1-y_i(t))}{y_i(t)}=\underset{t\to \infty }{\lim }\frac{1}{1-y_i(t)}>\rho$$

$$t\left(\frac{y_i(t)}{y_i(t+1)}-1\right)=t\frac{ϵu_i(\alpha )}{1-ϵu_i(\alpha )}.$$

$$C_t\doteq \left\{\exists {\alpha }^{\prime }\ne \alpha (t):x_{i{\alpha }_i^{\prime }}(t)>0,\text{mbox}foralli\in \mathcal{I}\right\},$$

$${\mathbb{P}}_z[A_{t+1}|A_t^c]$$

$$\underset{t\to \infty }{\mathrm{liminf}}{P}^t(z^{\ast },O)\ge \Pi (z^{\ast },O)>0.$$

$$P_{\lambda }=(1-\phi (\lambda ))P+\phi (\lambda )Q_{\lambda }$$

$$Q_{\lambda }=(1-\psi (\lambda ))Q+\psi (\lambda )Q^{\ast },$$

$$P_{\lambda }^L\doteq \phi (\lambda )\sum _{t=0}^{\infty }(1-\phi (\lambda ){)}^t{Q}_{\lambda }{P}^t=Q_{\lambda }{R}_{\lambda }$$

$$\Vert R_{\lambda }f-\Pi f{\Vert }_{\infty }$$

$$\phi (\lambda )\sum _{t=T}^{\infty }(1-\phi (\lambda ){)}^t\Vert P^{t}f-\Pi f{\Vert }_{\infty }$$

Full text

1 Introduction

Recently, multi-agent formulations have been utilized to tackle distributed optimization problems, since communication and computation complexity might be an issue in centralized optimization problems. In such formulations, decisions are usually taken in a repeated fashion, where agents select their next actions based on their own prior experience of the game.

The present paper discusses a class of reinforcement-learning dynamics, that belongs to the large family of Learning Automata [1, 2], within the context of (non-cooperative) strategic-form games. In this class of dynamics, agents are repeatedly involved in a game with a fixed payoff-matrix, and they need to decide which action to play next having only access to their own prior actions and payoffs. In Learning Automata, agents build their confidence over an action through repeated selection of this action and proportionally to the reward received from this action. Naturally, it has been utilized to analyze human-like (bounded) rationality [3].

Reinforcement learning has been applied in evolutionary economics, for modeling human and economic behavior [3, 4, 5, 6, 7]. It is also highly attractive to several engineering applications, since agents do not need to know neither the actions of the other agents, nor their own utility function. It has been utilized for system identification and pattern recognition [8], distributed network formation and coordination problems [9].

In strategic-form games, the main goal is to derive conditions under which convergence to Nash equilibria can be achieved. In social sciences, deriving such conditions may be important for justifying emergence of certain social phenomena. In engineering, convergence to Nash equilibria may also be desirable in distributed optimization problems, when the set of optimal solutions coincides with the set of Nash equilibria.

In Learning Automata, deriving conditions under which convergence to Nash equilibria is achieved may not be a trivial task. In particular, there are two main difficulties: a) excluding convergence to pure strategies that are not Nash equilibria, and b) excluding convergence to mixed strategy profiles. As it will be discussed in detail in a forthcoming Section 2, for some classes of (discrete-time) reinforcement-learning algorithms, convergence to non-Nash pure strategies may be achieved with positive probability. Moreover, excluding convergence to mixed strategy profiles may only be achieved under strong conditions in the utilities of the agents, (e.g., existence of a potential function).

In the present paper, we consider a class of (discrete-time) reinforcement-learning algorithms introduced in [9] that is closely related to existing algorithms for modeling human-like behavior, e.g., [3]. The main difference with prior reinforcement learning schemes lies in a) the step-size sequence, and b) the perturbation (or mutations) term. The step-size sequence is assumed constant, thus introducing a fading-memory effect of past experiences in each agent’s strategy. On the other hand, the perturbation term introduces errors in the selection process of each agent. Both these two features can be used for designing a desirable asymptotic behavior.

We provide an analytical framework for deriving conclusions over the asymptotic behavior of the dynamics that is based on an explicit characterization of the invariant probability measure of the induced Markov chain. In particular, we show that in all strategic-form games satisfying the Positive-Utility Property, the support of the invariant probability measure coincides with the set of pure strategy profiles. This extends prior work where nonconvergence to mixed strategy profiles may only be excluded under strong conditions in the payoff matrix (e.g., existence of a potential function). A detailed discussion of the exact contributions of this paper is provided in the forthcoming Section 2. At the end of the paper, we also provide a brief discussion over how the proposed framework can be further utilized to provide a more detailed characterization of the stochastically stable states (e.g., excluding convergence to non-Nash pure strategy profiles). Due to space limitations, this analysis is not presented in this paper.

In the remainder of the paper, Section 2 presents a class of reinforcement-learning dynamics, related work and the main contribution of this paper. Section 3 provides the main result of this paper (Theorem 3.1), where the set of stochastically stable states is characterized. A short discussion is also provided over the significance of this result and how it can be utilized to provide further conclusions. Finally, Section 4 provides the technical derivation of the main result and Section 5 presents concluding remarks.

Notation:

• $-$

For a Euclidean topological space $\mathcal{X}\subset\mathbb{R}^{n}$, let $\mathcal{N}_{\delta}(x)$ denote the $\delta$-neighborhood of $x\in\mathbb{R}^{n}$, i.e.,

[TABLE]

where $|\cdot|$ denotes the Euclidean distance.

• $-$

$e_{j}$ denotes the unit vector in $\mathbb{R}^{n}$ where its $j$th entry is equal to 1 and all other entries is equal to 0.

• $-$

$\Delta(n)$ denotes the probability simplex of dimension $n$, i.e.,

[TABLE]

• $-$

For some set $A$ in a topological space ${\mathcal{Z}}$, let $\mathbb{I}_{A}:{\mathcal{Z}}\to\{0,1\}$ denote the index function, i.e.,

[TABLE]

• $-$

$\delta_{x}$ denotes the Dirac measure at $x$.

• $-$

Let $A$ be a finite set and let any (finite) probability distribution $\sigma\in\Delta(\left|A\right|)$. The random selection of an element of $A$ will be denoted ${\rm rand}_{\sigma}[A]$. If $\sigma=(\nicefrac{{1}}{{\left|A\right|}},...,\nicefrac{{1}}{{\left|A\right|}})$, i.e., it corresponds to the uniform distribution, the random selection will be denoted by ${\rm rand}_{\rm unif}[A]$.

2 Reinforcement Learning

2.1 Terminology

We consider the standard setup of finite strategic-form games. Consider a finite set of agents (or players) $\mathcal{I}=\{1,...,n\}$, and let each agent have a finite set of actions $\mathcal{A}_{i}$. Let $\alpha_{i}\in\mathcal{A}_{i}$ denote any such action of agent $i$. The set of action profiles is the Cartesian product $\mathcal{A}\doteq\mathcal{A}_{1}\times\cdots\times\mathcal{A}_{n}$ and let $\alpha=(\alpha_{1},...,\alpha_{n})$ be a representative element of this set. We will denote $-i$ to be the complementary set $\mathcal{I}\backslash{i}$ and often decompose an action profile as follows $\alpha=(\alpha_{i},\alpha_{-i})$. The payoff/utility function of agent $i$ is a mapping $u_{i}(\cdot):\mathcal{A}\to\mathbb{R}$. A strategic-form game is defined by the triple $\langle{\mathcal{I},\mathcal{A},\{u_{i}(\cdot)\}_{i}}\rangle$.

For the remainder of the paper, we will be concerned with strategic-form games that satisfy the Positive-Utility Property.

Property 1 (Positive Utility Property)

For any agent $i\in\mathcal{I}$ and any action profile $\alpha\in\mathcal{A}$, $u_{i}(\alpha)>0$.

2.2 Reinforcement-learning algorithm

We consider a form of reinforcement learning that belongs to the general class of learning automata [2]. In learning automata, each agent updates a finite probability distribution $x_{i}\in\Delta(\left|\mathcal{A}_{i}\right|)$ representing its beliefs with respect to the most profitable action. The precise manner in which $x_{i}(t)$ changes at time $t$, depending on the performed action and the response of the environment, completely defines the reinforcement learning model.

The proposed reinforcement learning model is described in Table 1. At the first step, each agent $i$ updates its action given its current strategy vector $x_{i}(t)$. Its selection is slightly perturbed by a perturbation (or mutations) factor $\lambda>0$, such that, with a small probability $\lambda$ agent $i$ follows a uniform strategy (or, it trembles). At the second step, agent $i$ evaluates its new selection by collecting a utility measurement, while in the last step, agent $i$ updates its strategy vector given its new experience.

Here we identify actions $\mathcal{A}_{i}$ with vertices of the simplex, $\{e_{1},...,e_{\left|\mathcal{A}_{i}\right|}\}$. For example, if agent $i$ selects its $j$th action at time $t$, then $e_{\alpha_{i}(t)}\equiv e_{j}$. Note that by letting the step-size $\epsilon$ to be sufficiently small and since the utility function $u_{i}(\cdot)$ is uniformly bounded in $\mathcal{A}$, $x_{i}(t)\in\Delta(\left|\mathcal{A}_{i}\right|)$ for all $t$.

In case $\lambda=0$, the above update recursion will be referred to as the unperturbed reinforcement learning.

2.3 Related work

Erev-Roth type dynamics

In prior reinforcement learning in games, analysis has been restricted to decreasing step-size sequences $\epsilon(t)$ and $\lambda=0$. More specifically, in [3], the step-size sequence of agent $i$ is $\epsilon_{i}(t)=1/(ct^{\nu}+u_{i}(\alpha(t+1))$ for some positive constant $c$ and for $0<\nu<1$ (in the place of the constant step size $\epsilon$ of (2)). A comparative model is also used by [6], with $\epsilon_{i}(t)=1/(V_{i}(t)+u_{i}(\alpha(t+1)))$, where $V_{i}(t)$ is the accumulated benefits of agent $i$ up to time $t$ which gives rise to an urn process [5]. Some similarities are also shared with the Cross’ learning model of [4], where $\epsilon(t)=1$ and $u_{i}(\alpha(t))\leq{1}$, and its modification presented in [10], where $\epsilon(t)$, instead, is assumed decreasing.

The main difference of the proposed reinforcement-learning algorithm (Table 1) lies in the perturbation parameter $\lambda>0$ which was first introduced and analyzed in [9]. A state-dependent perturbation term has also been investigated in [11]. The perturbation parameter may serve as an equilibrium selection mechanism, since it excludes convergence to non-Nash action profiles. It resolved one of the main issues of several (discrete-time) reinforcement-learning algorithms, that is the positive probability of convergence to non-Nash action profiles under some conditions in the payoff function and the step-size sequence.

This issue has also been raised by [12, 6]. Reference [12] considered the model by [3] and showed that convergence to non-Nash pure strategy profiles can be excluded as long as $c>u_{i}(\alpha)$ for all $i\in\mathcal{I}$ and $\nu=1$. On the other hand, convergence to non-Nash action profiles was not an issue with the urn model of [5] (as analyzed in [6]). However, the use of an urn-process type step-size sequence significantly reduces the applicability of the reinforcement learning scheme. In conclusion, the perturbation parameter $\lambda>0$ may serve as a design tool for reinforcing convergence to Nash equilibria without necessarily employing an urn-process type step-size sequence. For engineering applications this is a desirable feature.

Although excluding convergence to non-Nash pure strategies can be guaranteed by using $\lambda>0$, establishing convergence to pure Nash equilibria may still be an issue, since it further requires excluding convergence to mixed strategy profiles. As presented in [11], this can be guaranteed only under strong conditions in the payoff matrix. For example, as shown in [11, Proposition 8], excluding convergence to mixed strategy profiles requires a) the existence of a potential function, b) conditions over the second gradient of the potential function. Requiring the existence of a potential function considerably restricts the class of games where equilibrium selection can be described. Furthermore, condition (b) may not easily be verified in games of large number of players or actions.

Learning automata

Certain forms of learning automata have been shown to converge to Nash equilibria in some classes of strategic-form games. For example, in [2], and for a generalized nonlinear reward-inaction scheme, convergence to Nash equilibrium strategies can be shown in identical interest games. Similar are the results presented in [13] for a linear reward-inaction scheme. These convergence results are restricted to games of payoffs in $[0,1]$. Extension to a larger class of games is possible if absolute monotonicity (cf., [2, Definition 8.1]) is shown (similarly to the discussion in [11, Proposition 8]).

Reference [14] introduced a class of linear reward-inaction schemes in combination with a coordinated exploration phase so that convergence to the efficient Nash equilibrium is achieved. However, coordination of the exploration phase requires communication between the players.

Recently, work by the author [9] has introduced a new class of learning automata (namely, perturbed learning automata) which can be applied in games with no restriction in the payoff matrix. Furthermore, a small perturbation factor also influences the decisions of the players, through which convergence to non-Nash pure strategy profiles can be excluded. However, to demonstrate global convergence, a monotonicity condition still needs to be established [11].

$Q$-learning

Similar questions of convergence to Nash equilibria also appear in alternative reinforcement learning formulations, such as approximate dynamic programming methodologies and $Q$-learning. However, this is usually accomplished under a stronger set of assumptions, which increases the computational complexity of the dynamics. For example, the Nash-Q learning algorithm of [15] addresses the problem of maximizing the discounted expected rewards for each agent by updating an approximation of the cost-to-go function (or $Q$-values). Alternative objectives may be used, such as the minimax criterion of [16]. However, it is indirectly assumed that agents need to have full access to the joint action space and the rewards received by the other agents.

More recently, reference [17] introduces a $Q$-learning scheme in combination with either adaptive play or better-reply dynamics in order to attain convergence to Nash equilibria in potential games [18] or weakly-acyclic games. However, this form of dynamics require that each player observes the actions selected by the other players, since a $Q$-value needs to be assigned in each joint action.

When the evaluation of the $Q$-values is totally independent, as in the individual $Q$-learning in [19], then convergence to Nash equilibria has been shown only for 2-player zero-sum games and 2-player partnership games with countably many Nash equilibria. Currently, there are no convergent results in games in multi-player games.

Payoff-based learning

The aforementioned types of dynamics can be considered as a form of payoff-based learning dynamics, since adaptation is only governed by the perceived utility of the players. Recently, there have been several attempts for establishing convergence to Nash equilibria through alternative payoff-based learning dynamics, (see, e.g., the benchmark-based dynamics of [20], or the aspiration-based dynamics in [21]). For these type of dynamics, convergence to Nash equilibria can be established without requiring any strong monotonicity property (e.g., in multi-player weakly-acyclic games in [20]). However, an investigation is required with respect to the resulting convergence rates as compared to the dynamics incorporating policy iterations (e.g., the Erev-Roth type of dynamics or the learning automata discussed above).

2.4 Objective

This paper provides an analytical framework for analyzing convergence in multi-player strategic-form games when players implement a class of perturbed learning-automata. We wish to impose no strong monotonicity assumptions in the structure of the game (e.g., the existence of a potential function). We provide a characterization of the invariant probability measure of the induced Markov chain that shows that only the pure-strategy profiles belong to its support. Thus, we implicitly exclude convergence to any mixed strategy profile (including mixed Nash equilibria). This result imposes no restrictions in the payoff matrix other than the Positive-Utility Property.

3 Convergence Analysis

3.1 Terminology and notation

Let ${\mathcal{Z}}\doteq\mathcal{A}\times\mathbf{\Delta}$, where $\mathbf{\Delta}\doteq\Delta(\left|\mathcal{A}_{1}\right|)\times\ldots\times\Delta(\left|\mathcal{A}_{n}\right|)$, i.e., pairs of joint actions $\alpha$ and nominal strategy profiles $x$. The set $\mathcal{A}$ is endowed with the discrete topology, $\mathbf{\Delta}$ with its usual Euclidean topology, and ${\mathcal{Z}}$ with the corresponding product topology. We also let ${\mathfrak{B}}({\mathcal{Z}})$ denote the Borel $\sigma$-field of ${\mathcal{Z}}$, and $\mathfrak{P}({\mathcal{Z}})$ the set of probability measures on ${\mathfrak{B}}({\mathcal{Z}})$ endowed with the Prohorov topology, i.e., the topology of weak convergence. The algorithm introduced in Table 1 defines an ${\mathcal{Z}}$-valued Markov chain. Let $P_{\lambda}:{\mathcal{Z}}\times{\mathfrak{B}}({\mathcal{Z}})\to[0,1]$ denote its transition probability function (t.p.f.), parameterized by $\lambda>0$. We refer to the process with $\lambda>0$ as the perturbed process. Let also $P:{\mathcal{Z}}\times{\mathfrak{B}}({\mathcal{Z}})$ denote the t.p.f. of the unperturbed process, i.e., when $\lambda=0$.

We let $C_{b}({\mathcal{Z}})$ denote the Banach space of real-valued continuous functions on ${\mathcal{Z}}$ under the sup-norm (denoted by $\|\cdot\|_{\infty}$) topology. For $f\in C_{b}({\mathcal{Z}})$, define

[TABLE]

and

[TABLE]

The process governed by the unperturbed process $P$ will be denoted by $\{Z_{t}:t\geq 0\}$. Let $\Omega\doteq{\mathcal{Z}}^{\infty}$ denote the canonical path space, i.e., an element $\omega\in\Omega$ is a sequence $\{\omega(0),\omega(1),\dotsc\}$, with $\omega(t)=(\alpha(t),x(t))\in{\mathcal{Z}}$. We use the same notation for the elements $(\alpha,x)$ of the space ${\mathcal{Z}}$ and for the coordinates of the process $Z_{t}=(\alpha(t),x(t))$. Let also ${\mathbb{P}}_{z}[\cdot]$ denote the unique probability measure induced by the unperturbed process $P$ on the product $\sigma$-algebra of ${\mathcal{Z}}^{\infty}$, initialized at $z=(\alpha,x)$, and ${\mathbb{E}}_{z}[\cdot]$ the corresponding expectation operator. Let also ${\mathfrak{F}}_{t}$, $t\geq{0}$, denote the $\sigma$-algebra generated by $\{Z_{\tau},~{}\tau\leq{t}\}$.

3.2 Stochastic stability

First, we note that both $P$ and $P_{\lambda}$ ($\lambda>0$) satisfy the weak Feller property (cf., [22, Definition 4.4.2]).

Proposition 1

Both the unperturbed process $P$ ($\lambda=0$) and the perturbed process $P_{\lambda}$ ($\lambda>0$) have the weak Feller property.

Proof

Let us consider any sequence $\{Z^{(k)}=(\alpha^{(k)},x^{(k)})\}$ such that $Z^{(k)}\to{Z}=(\alpha,x)\in{\mathcal{Z}}$.

For the unperturbed process governed by $P(\cdot,\cdot)$, and for any open set $O\in{\mathfrak{B}}({\mathcal{Z}})$, the following holds:

[TABLE]

where $\mathcal{P}_{\mathcal{X}_{i}}(O)$ and $\mathcal{P}_{\mathcal{A}}(O)$ are the canonical projections defined by the product topology. Similarly, we have:

[TABLE]

(a) Consider the case $x\in\mathbf{\Delta}^{o}$, i.e., $x$ belongs to the interior of $\mathbf{\Delta}$. For all $i\in\mathcal{I}$, due to the continuity of $\mathcal{R}_{i}(\cdot,\cdot)$ with respect to its second argument, and the fact that $O$ is an open set, there exists $\delta>0$ such that $\mathbb{I}_{\mathcal{P}_{\mathcal{X}_{i}}(O)}(\mathcal{R}_{i}(\alpha,x_{i}))=\mathbb{I}_{\mathcal{P}_{\mathcal{X}_{i}}(O)}(\alpha,y_{i}))$ for all $y_{i}\in\mathcal{N}_{\delta}(x_{i})$. Thus, for any sequence $Z^{(k)}=(\alpha^{(k)},x^{(k)})$ such that $Z^{(k)}\to Z=(\alpha,x)$, we have that $P(Z^{(k)},O)\to P(Z,O),$ as $k\to\infty$.

(b) Consider the case $x\in\partial\mathbf{\Delta}$, i.e., $x$ belongs to the boundary of $\mathbf{\Delta}$. Then, there exists $i\in\mathcal{I}$ such that $x_{i}\in\partial\Delta(\left|\mathcal{A}_{i}\right|)$, i.e., there exists an action $j\in{\mathcal{A}}_{i}$ such that $x_{ij}=0$. For any open set $O\in{\mathfrak{B}}({\mathcal{Z}})$, $x_{i}\notin\mathcal{P}_{\mathcal{X}_{i}}(O)$. Furthermore, for any $\alpha_{i}\in{\rm rand}_{x_{i}}[{\mathcal{A}}_{i}]$, $\mathbb{I}_{\mathcal{P}_{\mathcal{X}_{i}}(O)}(\mathcal{R}_{i}((\alpha_{i},\alpha_{-i}),x_{i})=0$ (since $x_{ij}=0$ and therefore $x_{i}$ cannot escape from the boundary). This directly implies that $P(Z=(\alpha,x),O)=0$. Construct a sequence $(\alpha^{(k)},x^{(k)})$ that converges to $(\alpha,x)$ such that $\alpha^{(k)}=\alpha$, $x_{i\alpha_{i}}^{(k)}>0$ and $x_{i}=e_{\alpha_{i}}$, i.e., the strategy of player $i$ converges to the vertex of action $\alpha_{i}$. Pick also $O\in{\mathfrak{B}}({\mathcal{Z}})$, such that $\mathbb{I}_{\mathcal{P}_{\mathcal{X}_{i}}(O)}(\mathcal{R}_{i}(\alpha,x_{i}^{(k)}))=1$ for all large $k$. This is always possible by selecting an open set $O$ such that $x\in\partial\mathcal{P}_{\mathcal{X}}(O)$ and $x^{(k)}\in\mathcal{P}_{\mathcal{X}}(O)$ for all $k$. In this case, $\lim_{k\to\infty}P(Z^{(k)},O)=1$. We conclude that for any sequence $Z^{(k)}=(\alpha^{(k)},x^{(k)})$ that converges to $Z=(\alpha,x)$, such that $x\in\partial\mathbf{\Delta}$, and for any open set $O\in{\mathfrak{B}}({\mathcal{Z}})$,

[TABLE]

By [22, Proposition 7.2.1], we conclude that $P$ satisfies the weak Feller property. The same steps can be followed to show that $P_{\lambda}$ also satisfies the weak Feller property.  $\bullet$

The measure $\mu_{\lambda}\in\mathfrak{P}({\mathcal{Z}})$ is called an invariant probability measure for $P_{\lambda}$ if

[TABLE]

Since ${\mathcal{Z}}$ defines a locally compact separable metric space and $P$, $P_{\lambda}$ have the weak Feller property, they both admit an invariant probability measure, denoted $\mu$ and $\mu_{\lambda}$, respectively [22, Theorem 7.2.3].

We would like to characterize the stochastically stable states $z\in{\mathcal{Z}}$ of $P_{\lambda}$, that is any state $z\in{\mathcal{Z}}$ for which any collection of invariant probability measures $\{\mu_{\lambda}\in\mathfrak{P}({\mathcal{Z}}):\mu_{\lambda}P_{\lambda}=\mu_{\lambda},\lambda>0\}$ satisfies $\liminf_{\lambda\to{0}}\mu_{\lambda}(z)>0$. As the forthcoming analysis will show, the stochastically stable states will be a subset of the set of pure strategy states (p.s.s.) defined as follows:

Definition 1 (Pure Strategy State)

*A pure strategy state is a state $s=(\alpha,x)\in{\mathcal{Z}}$ such that for all $i\in\mathcal{I}$, $x_{i}=e_{\alpha_{i}}$, i.e., $x_{i}$ coincides with the vertex of the probability simplex $\Delta(\left|\mathcal{A}_{i}\right|)$ which assigns probability 1 to action $\alpha_{i}$. *

We will denote the set of pure strategy states by $\mathcal{S}$.

Theorem 3.1 (Stochastic Stability)

There exists a unique probability vector $\pi=(\pi_{1},...,\pi_{\left|\mathcal{S}\right|})$ such that for any collection of invariant probability measures $\{\mu_{\lambda}\in\mathfrak{P}({\mathcal{Z}}):\mu_{\lambda}P_{\lambda}=\mu_{\lambda},\lambda>0\}$, the following hold:

• (a)

$\lim_{\lambda\to{0}}\mu_{\lambda}(\cdot)=\hat{\mu}(\cdot)\doteq\sum_{s\in\mathcal{S}}\pi_{s}\delta_{s}(\cdot),$* where convergence is in the weak sense.*

• (b)

The probability vector $\pi$ is an invariant distribution of the (finite-state) Markov process $\hat{P}$, such that, for any $s,s^{\prime}\in\mathcal{S}$,

[TABLE]

for any $\delta>0$ sufficiently small, where $Q$ is the t.p.f. corresponding to only one player trembling (i.e., following the uniform distribution of (1)).

The proof of Theorem 3.1 requires a series of propositions and will be presented in detail in Section 4.

3.3 Discussion

Theorem 3.1 establishes an important observation. That is, the “equivalence” (in a weak convergence sense) of the original (perturbed) learning process with a simplified process, where agents simultaneously tremble at the first iteration and then they do not tremble. This form of simplification of the dynamics has originally been exploited to analyze aspiration learning dynamics in [21], and it is based upon the fact that under the unperturbed dynamics, agents’ strategies will eventually converge to a pure strategy profile.

Furthermore, the limiting behavior of the original (perturbed) dynamics can be characterized by the (unique) invariant distribution of a finite-state Markov chain $\{P_{ss^{\prime}}\}$, whose states correspond to the pure-strategy states of the game. In other words, we should expect that as the perturbation parameter $\lambda$ approaches zero, the algorithm spends the majority of the time on pure strategy profiles. The importance of this result lies on the fact that no constraints have been imposed in the payoff matrix of the game other than the Positive-Utility Property 1. Thus, it extends to games beyond the fine set of potential games.

This convergence result can further be augmented with an ODE analysis for stochastic approximations to exclude convergence to pure strategies that are not Nash equilibria (as derived in [11] for the case of diminishing step size). Due to space limitations this analysis is not presented in this paper, however it can be the subject of future work.

4 Technical Derivation

4.1 Unperturbed Process

For $t\geq 0$ define the sets

[TABLE]

Note that $\{B_{t}:t\geq 0\}$ is a non-increasing sequence, i.e., $B_{t+1}\subseteq B_{t}$, while $\{A_{t}:t\geq 0\}$ is non-decreasing, i.e., $A_{t+1}\supseteq A_{t}$. Let

[TABLE]

In other words, the set $A_{\infty}$ corresponds to the event that agents eventually play the same action profile, while $B_{\infty}$ corresponds to the event that agents never change their actions.

Proposition 2 (Convergence to p.s.s.)

Let us assume that the step size $\epsilon>0$ is sufficiently small such that $0<\epsilon u_{i}(\alpha)<1$ for all $\alpha\in\mathcal{A}$ and for all agents $i\in\mathcal{I}$. Then, the following hold:

• (a)

$\inf_{z\in{\mathcal{Z}}}\;\mathbb{P}_{z}[B_{\infty}]>0$,

• (b)

$\inf_{z\in{\mathcal{Z}}}{\mathbb{P}}_{z}[A_{\infty}]=1$.

The first statement of Proposition 2 states that the probability that agents never change their actions is bounded away from zero, while the second statement states that the probability that eventually agents play the same action profile is one.

Proof

(a) Let us consider an action profile $\alpha=(\alpha_{1},...,\alpha_{n})\in\mathcal{A}$, and an initial strategy profile $x(0)=(x_{1}(0),...,x_{n}(0))$ such that $x_{i\alpha_{i}}(0)>0$ for all $i\in\mathcal{I}$. Note that if the same action profile $\alpha$ is selected up to time $t$, then the strategy of agent $i$ satisfies:

[TABLE]

Given that $B_{t}$ is non-increasing, from continuity from above we have

[TABLE]

Note that ${\mathbb{P}}[B_{\infty}]>0$ if and only if

[TABLE]

Let us introduce the variable

[TABLE]

which corresponds to the probability of agent $i$ selecting any action other than $\alpha_{i}$. Condition (7) is equivalent to

[TABLE]

We also have that

[TABLE]

for some $\rho>0$, since $0\leq y_{i}(t)\leq 1$. Thus, from the Limit Comparison Test, we conclude that condition (8) holds if and only if $\sum_{t=1}^{\infty}y_{i}(t)<\infty$, for each $i\in\mathcal{I}$.

Lastly, note that $y_{i}(t+1)/y_{i}(t)=1-\epsilon u_{i}(\alpha)$. By Raabe’s criterion, the series $\sum_{t=0}^{\infty}y_{i}(t)$ is convergent if $\lim_{t\to\infty}t\left(\nicefrac{{y_{i}(t)}}{{y_{i}(t+1)}}-1\right)>1.$ We have

[TABLE]

Thus, if $\epsilon u_{i}(\alpha)<1$ for all $\alpha\in\mathcal{A}$ and $i\in\mathcal{I}$, then $1-\epsilon u_{i}(\alpha)>0$ and $\lim_{t\to\infty}t(\nicefrac{{\epsilon u_{i}(\alpha)}}{{1-\epsilon u_{i}(\alpha)}})>1,$ which implies that the series $\sum_{t=1}^{\infty}y_{i}(t)$ is convergent. Thus, we conclude that ${\mathbb{P}}_{z}[B_{\infty}]>0$.

(b) Define the event

[TABLE]

i.e., $C_{t}$ corresponds to the event that there exists an action profile different from the current action profile for which the nominal strategy assigns positive probability for all agents $i$. Note that $A_{t}^{c}\subseteq C_{t}$, since $A_{t}^{c}$ occurs only if there is some action profile $\alpha^{\prime}\neq\alpha(t)$ for which the nominal strategy assigns positive probability. This further implies that ${\mathbb{P}}_{z}[A_{t}^{c}]\leq{\mathbb{P}}_{z}[C_{t}]$. Then, we have:

[TABLE]

where $c_{i}(\alpha^{\prime})\doteq 1-x_{i\alpha_{i}^{\prime}}(t)\geq 0$. We have already shown in part (a) that the second part of the r.h.s. is bounded away from zero. Therefore, we conclude that ${\mathbb{P}}_{z}[A_{t+1}|A_{t}^{c}]>0$. Thus, from the counterpart of the Borel-Cantelli Lemma, ${\mathbb{P}}_{z}\left[A_{\infty}\right]=1.$  $\bullet$

The above proposition is rather useful in characterizing the support of any invariant measure of the unperturbed process, as the following proposition shows.

Proposition 3 (Limiting t.p.f. of unperturbed process)

Let $\mu$ denote an invariant probability measure of $P$. Then, there exists a t.p.f. $\Pi$ on ${\mathcal{Z}}\times{\mathfrak{B}}({\mathcal{Z}})$ such that

• (a)

for $\mu$-a.e. $z\in{\mathcal{Z}}$, $\Pi(z,\cdot)$ is an invariant probability measure for $P$;

• (b)

for all $f\in C_{b}({\mathcal{Z}})$, $\lim_{t\to\infty}\|P^{t}f-\Pi f\|_{\infty}=0$;

• (c)

$\mu$* is an invariant probability measure of $\Pi$;*

• (d)

the support111The support of a measure $\mu$ on ${\mathcal{Z}}$ is the unique closed set $F\subset{\mathfrak{B}}({\mathcal{Z}})$ such that $\mu({\mathcal{Z}}\backslash{F})=0$ and $\mu(F\cap{O})>0$ for every open set $O\subset{\mathcal{Z}}$ such that $F\cap{O}\neq\varnothing$.* of $\Pi$ is on $\mathcal{S}$ for all $z\in{\mathcal{Z}}$.*

Proof

The state space ${\mathcal{Z}}$ is a locally compact separable metric space and the t.p.f. of the unperturbed process $P$ admits an invariant probability measure due to Proposition 1. Thus, statements (a), (b) and (c) follow directly from [22, Theorem 5.2.2 (a), (b), (e)].

(d) Let us assume that the support of $\Pi$ includes points in ${\mathcal{Z}}$ other than the pure strategy states. Let also $O\subset{\mathcal{Z}}$ be an open set such that $O\cap{\mathcal{S}}=\varnothing$ and $\Pi(z^{*},O)>0$ for some $z^{*}\in{\mathcal{Z}}$. Given that $P^{t}$ converges weakly to $\Pi$ as $t\to\infty$, from Portmanteau theorem (cf., [22, Theorem 1.4.16]), we have that

[TABLE]

This is a contradiction of Proposition 2(b). Thus, the conclusion follows.  $\bullet$

Proposition 3 states that the limiting unperturbed t.p.f. converges weakly to a t.p.f. $\Pi$ which accepts the same invariant p.m. as $P$. Furthermore, the support of $\Pi$ is the set of pure strategy states $\mathcal{S}$. This is a rather important observation, since the limiting perturbed process can also be “related” (in a weak-convergence sense) to the t.p.f. $\Pi$, as it will be shown in the following section.

4.2 Decomposition of perturbed t.p.f.

We can decompose the t.p.f. of the perturbed process as follows:

[TABLE]

where $\varphi(\lambda)=1-(1-\lambda)^{n}$ is the probability that at least one agent trembles (since $(1-\lambda)^{n}$ is the probability that no agent trembles), and $Q_{\lambda}$ corresponds to the t.p.f. induced by the one-step reinforcement-learning update when at least one agent trembles. Note that $\varphi(\lambda)\to{0}$ as $\lambda\to{0}$.

Define also $Q$ to be the t.p.f. when only one players trembles, and $Q^{*}$ is the t.p.f. where at least two players tremble. Then, we may write:

[TABLE]

where $\psi(\lambda)\doteq 1-\frac{n\lambda}{1-(1-\lambda)^{n}}$ corresponds to the probability that at least two players tremble given that at least one player trembles.

Let us also define the infinite-step t.p.f. when trembling only at the first step (briefly, lifted t.p.f.) as follows:

[TABLE]

where $R_{\lambda}\doteq\varphi(\lambda)\sum_{t=0}^{\infty}(1-\varphi(\lambda))^{t}P^{t},$ i.e., $R_{\lambda}$ corresponds to the resolvent t.p.f.

Proposition 4 (Invariant p.m. of perturbed process)

The following hold:

• (a)

For $f\in C_{b}({\mathcal{Z}})$, $\lim_{\lambda\to{0}}\|R_{\lambda}f-\Pi{f}\|_{\infty}=0.$

• (b)

For $f\in C_{b}({\mathcal{Z}})$, $\lim_{\lambda\to{0}}\|P_{\lambda}^{L}f-Q\Pi{f}\|_{\infty}=0$.

• (c)

Any invariant distribution $\mu_{\lambda}$ of $P_{\lambda}$ is also an invariant distribution of $P_{\lambda}^{L}$.

• (d)

Any weak limit point in $\mathfrak{P}({\mathcal{Z}})$ of $\mu_{\lambda}$, as $\lambda\to{0}$, is an invariant probability measure of $Q\Pi$.

Proof

(a) For any $f\in C_{b}({\mathcal{Z}})$, we have

[TABLE]

where we have used the property $\varphi(\lambda)\sum_{t=0}^{\infty}(1-\varphi(\lambda))^{t}=1$. Note that

[TABLE]

From Proposition 3(b), we have that for any $\delta>0$, there exists $T=T(\delta)>0$ such that the r.h.s. is uniformly bounded by $\delta$ for all $t\geq T$. Thus, the sequence

[TABLE]

is Cauchy and therefore convergent (under the sup-norm). In other words, there exists $A\in\mathbb{R}$ such that

[TABLE]

For every $T>0$, we have

[TABLE]

Note that

[TABLE]

If we take $\lambda\downarrow{0}$, then the r.h.s. converges to zero. Thus,

[TABLE]

which concludes the proof.

(b) For any $f\in C_{b}({\mathcal{Z}})$, we have

[TABLE]

The first term of the r.h.s. approaches 0 as $\lambda\downarrow{0}$ according to (a). The second term of the r.h.s. also approaches 0 as $\lambda\downarrow{0}$ since $Q_{\lambda}\rightarrow{Q}$ as $\lambda\downarrow{0}$.

(c) Note that, by definition of the perturbed t.p.f. $P_{\lambda}$, we have

[TABLE]

Note further that $Q_{\lambda}R_{\lambda}=P_{\lambda}^{L}$ and

[TABLE]

where $I$ corresponds to the identity operator. Thus, we have

[TABLE]

For any invariant probability measure of $P_{\lambda}$, $\mu_{\lambda}$, we have

[TABLE]

which equivalently implies that

[TABLE]

since $\mu_{\lambda}P_{\lambda}=\mu_{\lambda}$. Thus, we conclude that $\mu_{\lambda}$ is also an invariant p.m. of $P_{\lambda}^{L}$.

(d) Let $\hat{\mu}$ denote a weak limit point of $\mu_{\lambda}$ as $\lambda\downarrow{0}$. To see that such a limit exists, take $\hat{\mu}$ to be an invariant probability measure of $P$. Then,

[TABLE]

Note that the weak convergence of $P_{\lambda}$ to $P$, it necessarily implies that $\mu_{\lambda}\Rightarrow\hat{\mu}$. Note further that

[TABLE]

The first and the third term of the r.h.s. approaches 0 as $\lambda\downarrow{0}$ due to the fact that $\mu_{\lambda}\Rightarrow\hat{\mu}$. The same holds for the second term of the r.h.s. due to part (b). Thus, we conclude that any weak limit point of $\mu_{\lambda}$ as $\lambda\downarrow{0}$ is an invariant p.m. of $Q\Pi$.  $\bullet$

4.3 Invariant p.m. of one-step perturbed process

Define the finite-state Markov process $\hat{P}$ as in (5).

Proposition 5 (Unique invariant p.m. of $Q\Pi$)

There exists a unique invariant probability measure $\hat{\mu}$ of $Q\Pi$. It satisfies

[TABLE]

for some constants $\pi_{s}\geq{0}$, $s\in\mathcal{S}$. Moreover, $\pi=(\pi_{1},...,\pi_{\left|\mathcal{S}\right|})$ is an invariant distribution of $\hat{P}$, i.e., $\pi=\pi\hat{P}$.

Proof

From Proposition 3(d), we know that the support of $\Pi$ is on the set of pure strategy states $\mathcal{S}$. Thus, the support of $Q\Pi$ is also on $\mathcal{S}$. From Proposition 4, we know that $Q\Pi$ admits an invariant measure, say $\hat{\mu}$, whose support is also $\mathcal{S}$. Thus, $\hat{\mu}$ admits the form of (11), for some constants $\pi_{s}\geq{0}$, $s\in\mathcal{S}$.

Note also that $\mathcal{N}_{\delta}(s^{\prime})$ is a continuity set of $Q\Pi(s,\cdot)$, i.e., $Q\Pi(s,\partial\mathcal{N}_{\delta}(s^{\prime}))=0$. Thus, from Portmanteau theorem, given that $QP^{t}\Rightarrow Q\Pi$,

[TABLE]

If we also define $\pi_{s}\doteq\hat{\mu}(\mathcal{N}_{\delta}(s))$, then

[TABLE]

which shows that $\pi$ is an invariant distribution of $\hat{P}$, i.e., $\pi=\pi\hat{P}$.

It remains to establish uniqueness of the invariant distribution of $Q\Pi$. Note that the set $\mathcal{S}$ of pure strategy states is isomorphic with the set $\mathcal{A}$ of action profiles. If agent $i$ trembles (as t.p.f. $Q$ dictates), then all actions in $\mathcal{A}_{i}$ have positive probability of being selected, i.e., $Q(\alpha,(\alpha_{i}^{\prime},\alpha_{-i}))>0$ for all $\alpha_{i}^{\prime}\in\mathcal{A}_{i}$ and $i\in\mathcal{I}$. It follows by Proposition 2 that $Q\Pi(\alpha,(\alpha_{i}^{\prime},\alpha_{-i}))>0$ for all $\alpha_{i}^{\prime}\in\mathcal{A}_{i}$ and $i\in\mathcal{I}$. Finite induction then shows that $(Q\Pi)^{n}(\alpha,\alpha^{\prime})>0$ for all $\alpha,\alpha^{\prime}\in{\mathcal{A}}$. It follows that if we restrict the domain of $Q\Pi$ to $\mathcal{S}$, it defines an irreducible stochastic matrix. Therefore, $Q\Pi$ has a unique invariant distribution.  $\bullet$

4.4 Proof of Theorem 3.1

Theorem 3.1(a)–(b) is a direct implication of Propositions 4–5.

5 Conclusions & Future Work

In this paper, we considered a class of reinforcement-learning algorithms that belong to the family of learning automata, and we provided an explicit characterization of the invariant probability measure of its induced Markov chain. Through this analysis, we demonstrated convergence (in a weak sense) to the set of pure-strategy states, overcoming prior restrictions necessary under an ODE-approximation analysis, such as the existence of a potential function. Thus, we opened up new possibilities for equilibrium selection through this type of algorithms that goes beyond the fine class of potential games.

Although the set of pure-strategy-states (which are the stochastically-stable states) may contain non-Nash pure strategy profiles, a follow-up analysis that excludes convergence to such pure-strategy-states may be performed (similarly to the analysis presented in [11] for diminishing step size).