Low-Shot Validation: Active Importance Sampling for Estimating Classifier Performance on Rare Categories

TL;DR

This paper introduces a statistical validation method that accurately estimates classifier performance on rare categories with minimal labeled data, reducing annotation costs significantly.

Contribution

The authors develop a novel importance sampling-based validation algorithm that provides accurate F-score estimates in low-sample regimes, with a reliable variance estimator.

Findings

Achieves accurate F-score estimates with as few as 100 labels.

Reduces labeling effort by up to 10x compared to existing methods.

Effective on large-scale datasets like ImageNet and iNaturalist2017.

Abstract

For machine learning models trained with limited labeled training data, validation stands to become the main bottleneck to reducing overall annotation costs. We propose a statistical validation algorithm that accurately estimates the F-score of binary classifiers for rare categories, where finding relevant examples to evaluate on is particularly challenging. Our key insight is that simultaneous calibration and importance sampling enables accurate estimates even in the low-sample regime (< 300 samples). Critically, we also derive an accurate single-trial estimator of the variance of our method and demonstrate that this estimator is empirically accurate at low sample counts, enabling a practitioner to know how well they can trust a given low-sample estimate. When validating state-of-the-art semi-supervised models on ImageNet and iNaturalist2017, our method achieves the same estimates of model performance with up to 10x fewer labels than competing approaches. In particular, we can estimate model F1 scores with a variance of 0.005 using as few as 100 labels.