Beyond Static Bias: Adaptive Multi-Fidelity Bandits with Improving Proxies

TL;DR

This paper extends multi-fidelity bandit models to include improving proxies, proposing an adaptive algorithm that decides when to continue low-fidelity sampling or escalate to high-fidelity evaluations, with theoretical and empirical validation.

Contribution

It introduces a dynamic model for improving proxies in multi-fidelity bandits and proposes TACC, an adaptive algorithm with regret guarantees for cost-effective exploration.

Findings

TACC effectively balances low- and high-fidelity sampling in experiments.

Adaptive continuation reduces regret compared to fixed strategies.

Theoretical bounds demonstrate improved efficiency for intermediate arms.

Abstract

As an extension of the classical multi-armed bandit problem, multi-fidelity multi-armed bandits (MF-MAB) enable individual arms to be evaluated using diverse feedback sources that vary in both cost and accuracy. Prior stochastic models typically assume fixed low-to-high fidelity discrepancies, whereas modern proxy sources, such as learning-based simulators and Large Language Models (LLMs), can be improved using additional calibration. We investigate adaptive MF-MAB with improving proxy sources, and focus on the canonical two-fidelity case in which the low-fidelity source becomes more informative with repeated use. To capture this dynamic, we introduce a selected-average mismatch bound that converts dynamic low-fidelity observations into improvement-aware confidence bounds for the high-fidelity target. We propose the Threshold-Based Adaptive Continuation Companion (TACC), an optimistic algorithm that uses a bounded continuation rule to decide when low-fidelity sampling remains cost-effective and when to escalate. We prove an instance-dependent regret bound showing that, for detected intermediate arms, adaptive continuation replaces logarithmic high-fidelity confirmation with bounded low-fidelity continuation. Experiments on synthetic bandits and an LLM-as-a-judge policy-evaluation task examine when continuation improves cost-weighted regret.