Regression Oracles and Exploration Strategies for Short-Horizon Multi-Armed Bandits

TL;DR

This paper investigates multi-armed bandit strategies for very short interaction horizons, introducing regression oracles, new exploration patterns, and a parameter-free UCBT strategy, with experiments showing improved performance in game-like scenarios.

Contribution

It proposes the use of regression oracles and a new UCBT strategy for short-horizon bandit problems, addressing a less-studied setting with practical applications.

Findings

Regression oracles outperform simple averages in short horizons.

Epsilon-greedy with regression oracles achieves best results.

UCBT strategy performs well without tunable parameters.

Abstract

This paper explores multi-armed bandit (MAB) strategies in very short horizon scenarios, i.e., when the bandit strategy is only allowed very few interactions with the environment. This is an understudied setting in the MAB literature with many applications in the context of games, such as player modeling. Specifically, we pursue three different ideas. First, we explore the use of regression oracles, which replace the simple average used in strategies such as epsilon-greedy with linear regression models. Second, we examine different exploration patterns such as forced exploration phases. Finally, we introduce a new variant of the UCB1 strategy called UCBT that has interesting properties and no tunable parameters. We present experimental results in a domain motivated by exergames, where the goal is to maximize a player's daily steps. Our results show that the combination of epsilon-greedy or epsilon-decreasing with regression oracles outperforms all other tested strategies in the short horizon setting.