Consensus-Driven Active Model Selection

TL;DR

This paper introduces CODA, a probabilistic, consensus-driven active model selection method that efficiently identifies the best machine learning model with significantly less annotation effort by leveraging model disagreements.

Contribution

We develop CODA, a novel active model selection framework that models classifier relationships and uses Bayesian inference to reduce labeling costs in model selection tasks.

Findings

CODA reduces annotation effort by over 70% compared to previous methods.

It outperforms existing active model selection techniques across 26 benchmark tasks.

The probabilistic framework effectively captures model consensus and disagreement.

Abstract

The widespread availability of off-the-shelf machine learning models poses a challenge: which model, of the many available candidates, should be chosen for a given data analysis task? This question of model selection is traditionally answered by collecting and annotating a validation dataset -- a costly and time-intensive process. We propose a method for active model selection, using predictions from candidate models to prioritize the labeling of test data points that efficiently differentiate the best candidate. Our method, CODA, performs consensus-driven active model selection by modeling relationships between classifiers, categories, and data points within a probabilistic framework. The framework uses the consensus and disagreement between models in the candidate pool to guide the label acquisition process, and Bayesian inference to update beliefs about which model is best as more information is collected. We validate our approach by curating a collection of 26 benchmark tasks capturing a range of model selection scenarios. CODA outperforms existing methods for active model selection significantly, reducing the annotation effort required to discover the best model by upwards of 70% compared to the previous state-of-the-art. Code and data are available at https://github.com/justinkay/coda.