All models are wrong, some are useful: Model Selection with Limited Labels

TL;DR

This paper presents MODEL SELECTOR, a label-efficient framework for selecting the best pretrained model for a target dataset by sampling highly informative examples, significantly reducing labeling costs.

Contribution

Introduces MODEL SELECTOR, a novel method for efficient model selection with limited labels, demonstrating substantial reductions in labeling costs across diverse datasets and models.

Findings

Reduces labeling cost by up to 94.15% for selecting the best model.

Achieves up to 72.41% cost reduction when selecting a near-best model.

Consistently outperforms baseline methods across 18 model collections and 16 datasets.

Abstract

We introduce MODEL SELECTOR, a framework for label-efficient selection of pretrained classifiers. Given a pool of unlabeled target data, MODEL SELECTOR samples a small subset of highly informative examples for labeling, in order to efficiently identify the best pretrained model for deployment on this target dataset. Through extensive experiments, we demonstrate that MODEL SELECTOR drastically reduces the need for labeled data while consistently picking the best or near-best performing model. Across 18 model collections on 16 different datasets, comprising over 1,500 pretrained models, MODEL SELECTOR reduces the labeling cost by up to 94.15% to identify the best model compared to the cost of the strongest baseline. Our results further highlight the robustness of MODEL SELECTOR in model selection, as it reduces the labeling cost by up to 72.41% when selecting a near-best model, whose accuracy is only within 1% of the best model.