Evaluating multiple models using labeled and unlabeled data

TL;DR

This paper introduces SSME, a semi-supervised evaluation method that leverages both labeled and unlabeled data to accurately assess classifiers, especially in domains with limited labeled datasets.

Contribution

The paper presents SSME, the first evaluation approach to utilize unlabeled data and multiple classifiers simultaneously for more accurate performance estimation.

Findings

SSME reduces evaluation error by 5.1x compared to using labeled data alone.

SSME outperforms existing methods in diverse domains like healthcare and image annotation.

It improves performance estimates across subgroups and for language models.

Abstract

It remains difficult to evaluate machine learning classifiers in the absence of a large, labeled dataset. While labeled data can be prohibitively expensive or impossible to obtain, unlabeled data is plentiful. Here, we introduce Semi-Supervised Model Evaluation (SSME), a method that uses both labeled and unlabeled data to evaluate machine learning classifiers. SSME is the first evaluation method to take advantage of the fact that: (i) there are frequently multiple classifiers for the same task, (ii) continuous classifier scores are often available for all classes, and (iii) unlabeled data is often far more plentiful than labeled data. The key idea is to use a semi-supervised mixture model to estimate the joint distribution of ground truth labels and classifier predictions. We can then use this model to estimate any metric that is a function of classifier scores and ground truth labels (e.g., accuracy or expected calibration error). We present experiments in four domains where obtaining large labeled datasets is often impractical: (1) healthcare, (2) content moderation, (3) molecular property prediction, and (4) image annotation. Our results demonstrate that SSME estimates performance more accurately than do competing methods, reducing error by 5.1x relative to using labeled data alone and 2.4x relative to the next best competing method. SSME also improves accuracy when evaluating performance across subsets of the test distribution (e.g., specific demographic subgroups) and when evaluating the performance of language models.