Aspiration-based Perturbed Learning Automata in Games with Noisy Utility Measurements. Part B: Stochastic Stability in Weakly Acyclic Games

TL;DR

This paper introduces aspiration-based perturbed learning automata (APLA), a novel reinforcement learning scheme that guarantees convergence to pure Nash equilibria in weakly-acyclic games with noisy observations, extending prior work to more general game classes.

Contribution

The paper proposes APLA, a new payoff-based learning scheme with stochastic stability analysis in noisy, multi-player weakly-acyclic games, addressing convergence limitations of previous methods.

Findings

Convergence to pure Nash equilibria is achieved under certain conditions.

APLA extends reinforcement learning to weakly-acyclic games with noisy data.

Simulation supports theoretical convergence results.

Abstract

Reinforcement-based learning dynamics may exhibit several limitations when applied in a distributed setup. In (repeatedly-played) multi-player/action strategic-form games, and when each player applies an independent copy of the learning dynamics, convergence to (usually desirable) pure Nash equilibria cannot be guaranteed. Prior work has only focused on a small class of games, namely potential and coordination games. Furthermore, strong convergence guarantees (i.e., almost sure convergence or weak convergence) are mostly restricted to two-player games. To address this main limitation of reinforcement-based learning in repeatedly-played strategic-form games, this paper introduces a novel payoff-based learning scheme for distributed optimization in multi-player/action strategic-form games. We present an extension of perturbed learning automata (PLA), namely aspiration-based perturbed learning automata (APLA), in which each player's probability distribution for selecting actions is reinforced both by repeated selection and an aspiration factor that captures the player's satisfaction level. We provide a stochastic stability analysis of APLA in multi-player positive-utility games under the presence of noisy observations. This paper is the second part of this study that analyzes stochastic stability in multi-player/action weakly-acyclic games in the presence of noisy observations. We provide conditions under which convergence is attained (in weak sense) to the set of pure Nash equilibria and payoff-dominant equilibria. To the best of our knowledge, this is the first reinforcement-based learning scheme that addresses convergence in weakly-acyclic games. Lastly, we provide a specialization of the results to the classical Stag-Hunt game, supported by a simulation study.