Pairwise Elimination with Instance-Dependent Guarantees for Bandits with Cost Subsidy

TL;DR

This paper introduces pairwise-elimination algorithms for bandit problems with cost constraints, providing instance-dependent guarantees and demonstrating their optimality and effectiveness through theoretical analysis and real-world datasets.

Contribution

It proposes the PE and PE-CS algorithms for cost-sensitive bandits with reward constraints, offering the first order-wise logarithmic regret guarantees and establishing their optimality.

Findings

PE and PE-CS achieve logarithmic upper bounds on regret.

PE is order-optimal for known reference arm instances.

Experiments show PE and PE-CS outperform baselines in real datasets.

Abstract

Multi-armed bandits (MAB) are commonly used in sequential online decision-making when the reward of each decision is an unknown random variable. In practice however, the typical goal of maximizing total reward may be less important than minimizing the total cost of the decisions taken, subject to a reward constraint. For example, we may seek to make decisions that have at least the reward of a reference ``default'' decision, with as low a cost as possible. This problem was recently introduced in the Multi-Armed Bandits with Cost Subsidy (MAB-CS) framework. MAB-CS is broadly applicable to problem domains where a primary metric (cost) is constrained by a secondary metric (reward), and the rewards are unknown. In our work, we address variants of MAB-CS including ones with reward constrained by the reward of a known reference arm or by the subsidized best reward. We introduce the Pairwise-Elimination (PE) algorithm for the known reference arm variant and generalize PE to PE-CS for the subsidized best reward variant. Our instance-dependent analysis of PE and PE-CS reveals that both algorithms have an order-wise logarithmic upper bound on Cost and Quality Regret, making our policies the first with such a guarantee. Moreover, by comparing our upper and lower bound results we establish that PE is order-optimal for all known reference arm problem instances. Finally, experiments are conducted using the MovieLens 25M and Goodreads datasets for both PE and PE-CS revealing the effectiveness of PE and the superior balance between performance and reliability offered by PE-CS compared to baselines from the literature.