Asymptotic properties of a multicolored random reinforced urn model with an application to multi-armed bandits

TL;DR

This paper analyzes the asymptotic behavior of a multicolored, multi-drawing reinforced urn model, establishing convergence properties and applying findings to hypothesis testing in multi-armed bandit problems.

Contribution

It introduces a multicolored, multi-drawing reinforced urn model and derives its limiting behavior, strong convergence estimators, and their asymptotic independence, with applications to multi-armed bandits.

Findings

Established strong convergence of urn composition.

Derived asymptotic normality of reinforcement mean estimators.

Identified asymptotic independence among estimators of different reinforcement means.

Abstract

The random self-reinforcement mechanism, characterized by the principle of ``the rich get richer'', has demonstrated significant utility across various domains. One prominent model embodying this mechanism is the random reinforcement urn model. This paper investigates a multicolored, multiple-drawing variant of the random reinforced urn model. We establish the limiting behavior of the normalized urn composition and demonstrate strong convergence upon scaling the counts of each color. Additionally, we derive strong convergence estimators for the reinforcement means, i.e., for the expectations of the replacement matrix's diagonal elements, and prove their joint asymptotic normality. It is noteworthy that the estimators of the largest reinforcement mean are asymptotically independent of the estimators of the other smaller reinforcement means. Additionally, if a reinforcement mean is not the largest, the estimators of these smaller reinforcement means will also demonstrate asymptotic independence among themselves. Furthermore, we explore the parallels between the reinforced mechanisms in random reinforced urn models and multi-armed bandits, addressing hypothesis testing for expected payoffs in the latter context.