Efficient Online Set-valued Classification with Bandit Feedback

TL;DR

This paper introduces BCCP, a novel method for set-valued classification in online bandit feedback settings, providing coverage guarantees despite limited label information and sparse data.

Contribution

The paper proposes BCCP, a new conformal prediction approach that works with bandit feedback, addressing label sparsity and providing class-specific coverage guarantees.

Findings

BCCP achieves reliable coverage guarantees in online bandit settings.

The method effectively handles sparse and partial label information.

Experimental results demonstrate improved set accuracy over existing methods.

Abstract

Conformal prediction is a distribution-free method that wraps a given machine learning model and returns a set of plausible labels that contain the true label with a prescribed coverage rate. In practice, the empirical coverage achieved highly relies on fully observed label information from data both in the training phase for model fitting and the calibration phase for quantile estimation. This dependency poses a challenge in the context of online learning with bandit feedback, where a learner only has access to the correctness of actions (i.e., pulled an arm) but not the full information of the true label. In particular, when the pulled arm is incorrect, the learner only knows that the pulled one is not the true class label, but does not know which label is true. Additionally, bandit feedback further results in a smaller labeled dataset for calibration, limited to instances with correct actions, thereby affecting the accuracy of quantile estimation. To address these limitations, we propose Bandit Class-specific Conformal Prediction (BCCP), offering coverage guarantees on a class-specific granularity. Using an unbiased estimation of an estimand involving the true label, BCCP trains the model and makes set-valued inferences through stochastic gradient descent. Our approach overcomes the challenges of sparsely labeled data in each iteration and generalizes the reliability and applicability of conformal prediction to online decision-making environments.