A Farewell to Arms: Sequential Reward Maximization on a Budget with a Giving Up Option

TL;DR

This paper introduces a new sequential decision-making model with a giving-up option, formulates it as a stochastic bandit problem with resource consumption, and proposes an algorithm with proven regret bounds.

Contribution

It formulates a novel bandit problem with giving-up options and develops WAIT-UCB, an algorithm with improved regret bounds over existing methods.

Findings

WAIT-UCB achieves logarithmic regret bounds.

Simulation results show WAIT-UCB outperforms state-of-the-art algorithms.

Optimal arm selection is based on reward-to-waiting-time ratio.

Abstract

We consider a sequential decision-making problem where an agent can take one action at a time and each action has a stochastic temporal extent, i.e., a new action cannot be taken until the previous one is finished. Upon completion, the chosen action yields a stochastic reward. The agent seeks to maximize its cumulative reward over a finite time budget, with the option of "giving up" on a current action -- hence forfeiting any reward -- in order to choose another action. We cast this problem as a variant of the stochastic multi-armed bandits problem with stochastic consumption of resource. For this problem, we first establish that the optimal arm is the one that maximizes the ratio of the expected reward of the arm to the expected waiting time before the agent sees the reward due to pulling that arm. Using a novel upper confidence bound on this ratio, we then introduce an upper confidence based-algorithm, WAIT-UCB, for which we establish logarithmic, problem-dependent regret bound which has an improved dependence on problem parameters compared to previous works. Simulations on various problem configurations comparing WAIT-UCB against the state-of-the-art algorithms are also presented.