Classifier Risk Estimation under Limited Labeling Resources

TL;DR

This paper introduces stratified sampling strategies to accurately estimate classifier performance with minimal labeled data, significantly reducing variance and sample size needed.

Contribution

It presents novel stratified sampling methods that outperform simple random sampling for classifier accuracy estimation under limited labeling resources.

Findings

Stratified sampling reduces estimation variance by over 65%.

Methods require up to 60% fewer samples for 1% accuracy error.

Strategies outperform random sampling in accuracy estimation.

Abstract

In this paper we propose strategies for estimating performance of a classifier when labels cannot be obtained for the whole test set. The number of test instances which can be labeled is very small compared to the whole test data size. The goal then is to obtain a precise estimate of classifier performance using as little labeling resource as possible. Specifically, we try to answer, how to select a subset of the large test set for labeling such that the performance of a classifier estimated on this subset is as close as possible to the one on the whole test set. We propose strategies based on stratified sampling for selecting this subset. We show that these strategies can reduce the variance in estimation of classifier accuracy by a significant amount compared to simple random sampling (over 65% in several cases). Hence, our proposed methods are much more precise compared to random sampling for accuracy estimation under restricted labeling resources. The reduction in number of samples required (compared to random sampling) to estimate the classifier accuracy with only 1% error is high as 60% in some cases.