Online Active Model Selection for Pre-trained Classifiers

TL;DR

This paper introduces an online active model selection method that efficiently chooses which data points to label in order to identify the best pre-trained classifier with minimal queries, applicable to both adversarial and stochastic data streams.

Contribution

It presents a novel online selective sampling algorithm with theoretical guarantees for model selection among pre-trained classifiers in streaming settings.

Findings

Algorithm achieves high probability of selecting the best model.

Effective in both adversarial and stochastic data streams.

Theoretical guarantees support practical performance.

Abstract

Given $k$ pre-trained classifiers and a stream of unlabeled data examples, how can we actively decide when to query a label so that we can distinguish the best model from the rest while making a small number of queries? Answering this question has a profound impact on a range of practical scenarios. In this work, we design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can be used for online prediction tasks for both adversarial and stochastic streams. We establish several theoretical guarantees for our algorithm and extensively demonstrate its effectiveness in our experimental studies.