Regulating Greed Over Time in Multi-Armed Bandits

TL;DR

This paper introduces time-aware modifications to multi-armed bandit algorithms to better handle predictable, time-dependent customer behavior patterns in retail, improving regret performance by regulating greed over time.

Contribution

It proposes new methods that incorporate known time-dependent patterns into bandit algorithms, including adaptations of UCB, ε-greedy, UCB-L, and a novel variable arm pool policy.

Findings

Corrected algorithms reduce regret in retail scenarios.

Time-dependent regulation of exploration and exploitation improves performance.

Experimental results outperform traditional bandit methods.

Abstract

In retail, there are predictable yet dramatic time-dependent patterns in customer behavior, such as periodic changes in the number of visitors, or increases in customers just before major holidays. The current paradigm of multi-armed bandit analysis does not take these known patterns into account. This means that for applications in retail, where prices are fixed for periods of time, current bandit algorithms will not suffice. This work provides a remedy that takes the time-dependent patterns into account, and we show how this remedy is implemented for the UCB, $\varepsilon$-greedy, and UCB-L algorithms, and also through a new policy called the variable arm pool algorithm. In the corrected methods, exploitation (greed) is regulated over time, so that more exploitation occurs during higher reward periods, and more exploration occurs in periods of low reward. In order to understand why regret is reduced with the corrected methods, we present a set of bounds that provide insight into why we would want to exploit during periods of high reward, and discuss the impact on regret. Our proposed methods perform well in experiments, and were inspired by a high-scoring entry in the Exploration and Exploitation 3 contest using data from Yahoo$!$ Front Page. That entry heavily used time-series methods to regulate greed over time, which was substantially more effective than other contextual bandit methods.