Understanding the stochastic dynamics of sequential decision-making processes: A path-integral analysis of multi-armed bandits

TL;DR

This paper applies statistical physics techniques to analyze the stochastic dynamics of multi-armed bandit models, revealing finite-time regret distributions and complex behaviors like multimodality and regret due to early exploitation errors.

Contribution

It introduces a novel analytical approach using path-integral methods to characterize finite-time regret distributions in MAB models, bridging decision-making and physics.

Findings

Finite-time regret distribution is multimodal.

Large regrets often stem from early sub-optimal exploitation.

Analytical results align well with simulations.

Abstract

The multi-armed bandit (MAB) model is one of the most classical models to study decision-making in an uncertain environment. In this model, a player chooses one of $K$ possible arms of a bandit machine to play at each time step, where the corresponding arm returns a random reward to the player, potentially from a specific unknown distribution. The target of the player is to collect as many rewards as possible during the process. Despite its simplicity, the MAB model offers an excellent playground for studying the trade-off between exploration versus exploitation and designing effective algorithms for sequential decision-making under uncertainty. Although many asymptotically optimal algorithms have been established, the finite-time behaviors of the stochastic dynamics of the MAB model appear much more challenging to analyze, due to the intertwine between the decision-making and the rewards being collected. In this paper, we employ techniques in statistical physics to analyze the MAB model, which facilitates the characterization of the distribution of cumulative regrets at a finite short time, the central quantity of interest in an MAB algorithm, as well as the intricate dynamical behaviors of the model. Our analytical results, in good agreement with simulations, point to the emergence of an interesting multimodal regret distribution, with large regrets resulting from excess exploitation of sub-optimal arms due to an initial unlucky output from the optimal one.