Risk averse non-stationary multi-armed bandits

TL;DR

This paper introduces methods for risk-averse decision-making in non-stationary multi-armed bandit problems using CVaR, proposing two estimation techniques and demonstrating their effectiveness through simulations.

Contribution

It presents two novel CVaR estimation methods for non-stationary bandits and integrates them into classic policies, improving risk-aware decision-making.

Findings

Proposed two CVaR estimation techniques for non-stationary losses.

Embedded these estimators into epsilon-greedy policies.

Showed improved performance over naive benchmarks in simulations.

Abstract

This paper tackles the risk averse multi-armed bandits problem when incurred losses are non-stationary. The conditional value-at-risk (CVaR) is used as the objective function. Two estimation methods are proposed for this objective function in the presence of non-stationary losses, one relying on a weighted empirical distribution of losses and another on the dual representation of the CVaR. Such estimates can then be embedded into classic arm selection methods such as epsilon-greedy policies. Simulation experiments assess the performance of the arm selection algorithms based on the two novel estimation approaches, and such policies are shown to outperform naive benchmarks not taking non-stationarity into account.