Regret bounds for Narendra-Shapiro bandit algorithms

TL;DR

This paper investigates the regret bounds of Narendra-Shapiro bandit algorithms, demonstrating that penalized versions can achieve sublinear regret bounds and extending convergence results to multi-armed cases with PDMP analysis.

Contribution

The paper provides the first regret bounds for penalized Narendra-Shapiro algorithms and extends convergence and ergodic properties to multi-armed bandits with PDMP analysis.

Findings

Pseudo-regret is bounded by C√n for penalized two-armed bandits.

Convergence results are extended to multi-armed bandits.

Ergodic properties of the associated PDMP are established.

Abstract

Narendra-Shapiro (NS) algorithms are bandit-type algorithms that have been introduced in the sixties (with a view to applications in Psychology or learning automata), whose convergence has been intensively studied in the stochastic algorithm literature. In this paper, we adress the following question: are the Narendra-Shapiro (NS) bandit algorithms competitive from a \textit{regret} point of view? In our main result, we show that some competitive bounds can be obtained for such algorithms in their penalized version (introduced in \cite{Lamberton_Pages}). More precisely, up to an over-penalization modification, the pseudo-regret $\bar{R}_n$ related to the penalized two-armed bandit algorithm is uniformly bounded by $C \sqrt{n}$ (where $C$ is made explicit in the paper). \noindent We also generalize existing convergence and rates of convergence results to the multi-armed case of the over-penalized bandit algorithm, including the convergence toward the invariant measure of a Piecewise Deterministic Markov Process (PDMP) after a suitable renormalization. Finally, ergodic properties of this PDMP are given in the multi-armed case.