Exploration vs Exploitation vs Safety: Risk-averse Multi-Armed Bandits

TL;DR

This paper introduces MARAB, a risk-aware multi-armed bandit algorithm that balances exploration, exploitation, and safety by incorporating conditional value at risk, with theoretical and empirical validation.

Contribution

It presents the MARAB algorithm for risk-averse bandit problems and provides a theoretical analysis of the MIN algorithm, demonstrating robustness compared to UCB.

Findings

MARAB effectively limits risky exploration in bandit problems.

Theoretical analysis shows MIN's robustness over UCB under mild assumptions.

Experimental results validate MARAB and MIN's performance on artificial and real-world data.

Abstract

Motivated by applications in energy management, this paper presents the Multi-Armed Risk-Aware Bandit (MARAB) algorithm. With the goal of limiting the exploration of risky arms, MARAB takes as arm quality its conditional value at risk. When the user-supplied risk level goes to 0, the arm quality tends toward the essential infimum of the arm distribution density, and MARAB tends toward the MIN multi-armed bandit algorithm, aimed at the arm with maximal minimal value. As a first contribution, this paper presents a theoretical analysis of the MIN algorithm under mild assumptions, establishing its robustness comparatively to UCB. The analysis is supported by extensive experimental validation of MIN and MARAB compared to UCB and state-of-art risk-aware MAB algorithms on artificial and real-world problems.