Multi-Armed Bandits With Best-Action Queries

TL;DR

This paper investigates the impact of best-action queries on multi-armed bandit problems under bandit feedback, providing a complete characterization of their benefits in stochastic and adversarial settings.

Contribution

It resolves whether best-action queries improve regret bounds in bandit feedback, showing they do in stochastic i.i.d. rewards but not in correlated rewards.

Findings

Best-action queries reduce regret to (((T/k,\,((T-k))) in stochastic i.i.d. rewards.

Any algorithm faces a ((((T-k))) regret lower bound in correlated rewards.

Full-feedback results do not extend to bandit feedback in correlated reward settings.

Abstract

We study \emph{multi-armed bandits} (MABs) augmented with \emph{best-action queries}, in which the learner may additionally query an oracle that reveals the best arm in the current round. This setting was recently characterized by Russo et al. [2024] in the \emph{full-feedback} model, where the learner observes the rewards of all arms after each round. They show that, in both \emph{stochastic} and \emph{adversarial} environments, $k$ best-action queries reduce the optimal $\widetilde{\mathcal{O}}(\sqrt{T})$ regret to $\widetilde{\mathcal{O}}(\min\{T/k,\sqrt{T}\})$. Whether this improvement extends to the more realistic \emph{bandit-feedback} model -- where the learner observes only the reward of the played arm -- was left as an open problem. We fully resolve this question. When rewards are stochastic but correlated among arms, we show that the full-feedback result does not carry over: any algorithm must incur regret at least $\Omega(\sqrt{T-k})$. This lower bound directly extends to adversarial environments. On the positive side, we show that $\widetilde{\mathcal{O}}(\min\{T/k,\sqrt{T-k}\})$ regret is still achievable when rewards are stochastic and i.i.d., and establish a matching lower bound, up to logarithmic factors. Together, these results provide a complete characterization of the benefits of \emph{best-action queries} in the \emph{bandit-feedback} model.