Using Subjective Logic to Estimate Uncertainty in Multi-Armed Bandit Problems

TL;DR

This paper introduces a novel approach using subjective logic to better estimate and manage uncertainty in multi-armed bandit problems, distinguishing between inherent randomness and limited knowledge.

Contribution

It proposes new algorithms based on subjective logic for multi-armed bandits and compares their performance with classical methods, providing insights into uncertainty evaluation.

Findings

Subjective logic enables effective uncertainty assessment.

New algorithms outperform classical methods in certain scenarios.

Insights into the dynamics of epistemic and aleatoric uncertainty.

Abstract

The multi-armed bandit problem is a classical decision-making problem where an agent has to learn an optimal action balancing exploration and exploitation. Properly managing this trade-off requires a correct assessment of uncertainty; in multi-armed bandits, as in other machine learning applications, it is important to distinguish between stochasticity that is inherent to the system (aleatoric uncertainty) and stochasticity that derives from the limited knowledge of the agent (epistemic uncertainty). In this paper we consider the formalism of subjective logic, a concise and expressive framework to express Dirichlet-multinomial models as subjective opinions, and we apply it to the problem of multi-armed bandits. We propose new algorithms grounded in subjective logic to tackle the multi-armed bandit problem, we compare them against classical algorithms from the literature, and we analyze the insights they provide in evaluating the dynamics of uncertainty. Our preliminary results suggest that subjective logic quantities enable useful assessment of uncertainty that may be exploited by more refined agents.