Contextual Active Model Selection

TL;DR

This paper introduces CAMS, a novel algorithm for active model selection that minimizes labeling costs by leveraging context information, with theoretical guarantees and strong empirical performance on benchmark datasets.

Contribution

The paper proposes CAMS, a new contextual active model selection algorithm with theoretical analysis and demonstrated effectiveness in reducing labeling costs on benchmarks.

Findings

CAMS achieves less than 10% labeling effort on CIFAR10 and DRIFT benchmarks.

CAMS maintains comparable or better accuracy than existing methods.

Theoretical analysis provides regret and query complexity bounds under various settings.

Abstract

While training models and labeling data are resource-intensive, a wealth of pre-trained models and unlabeled data exists. To effectively utilize these resources, we present an approach to actively select pre-trained models while minimizing labeling costs. We frame this as an online contextual active model selection problem: At each round, the learner receives an unlabeled data point as a context. The objective is to adaptively select the best model to make a prediction while limiting label requests. To tackle this problem, we propose CAMS, a contextual active model selection algorithm that relies on two novel components: (1) a contextual model selection mechanism, which leverages context information to make informed decisions about which model is likely to perform best for a given context, and (2) an active query component, which strategically chooses when to request labels for data points, minimizing the overall labeling cost. We provide rigorous theoretical analysis for the regret and query complexity under both adversarial and stochastic settings. Furthermore, we demonstrate the effectiveness of our algorithm on a diverse collection of benchmark classification tasks. Notably, CAMS requires substantially less labeling effort (less than 10%) compared to existing methods on CIFAR10 and DRIFT benchmarks, while achieving similar or better accuracy. Our code is publicly available at: https://github.com/xuefeng-cs/Contextual-Active-Model-Selection.