Softmax gradient policy for variance minimization and risk-averse multi armed bandits

TL;DR

This paper introduces a risk-aware multi-armed bandit algorithm that minimizes variance using a softmax policy, with proven convergence and practical insights from numerical experiments.

Contribution

It proposes a novel softmax-based algorithm for variance minimization in risk-averse bandits, with theoretical convergence guarantees and practical implementation guidance.

Findings

The algorithm converges under natural conditions.

Numerical experiments illustrate practical behavior.

Guidance on implementation choices is provided.

Abstract

Algorithms for the Multi-Armed Bandit (MAB) problem play a central role in sequential decision-making and have been extensively explored both theoretically and numerically. While most classical approaches aim to identify the arm with the highest expected reward, we focus on a risk-aware setting where the goal is to select the arm with the lowest variance, favoring stability over potentially high but uncertain returns. To model the decision process, we consider a softmax parameterization of the policy; we propose a new algorithm to select the minimal variance (or minimal risk) arm and prove its convergence under natural conditions. The algorithm constructs an unbiased estimate of the objective by using two independent draws from the current's arm distribution. We provide numerical experiments that illustrate the practical behavior of these algorithms and offer guidance on implementation choices. The setting also covers general risk-aware problems where there is a trade-off between maximizing the average reward and minimizing its variance.