Label-Efficient Monitoring of Classification Models via Stratified Importance Sampling

TL;DR

This paper introduces a stratified importance sampling framework for monitoring classification models efficiently under strict label budgets, providing unbiased estimators with improved accuracy over traditional sampling methods.

Contribution

The paper develops a general, theoretically grounded stratified importance sampling approach that enhances model monitoring efficiency without requiring optimal stratification or proposal distributions.

Findings

SIS yields unbiased estimators with lower MSE than IS and SRS.

Experimental results show consistent efficiency gains across various tasks.

SIS is practical and lightweight for real-world model monitoring.

Abstract

Monitoring the performance of classification models in production is critical yet challenging due to strict labeling budgets, one-shot batch acquisition of labels and extremely low error rates. We propose a general framework based on Stratified Importance Sampling (SIS) that directly addresses these constraints in model monitoring. While SIS has previously been applied in specialized domains, our theoretical analysis establishes its broad applicability to the monitoring of classification models. Under mild conditions, SIS yields unbiased estimators with strict finite-sample mean squared error (MSE) improvements over both importance sampling (IS) and stratified random sampling (SRS). The framework does not rely on optimally defined proposal distributions or strata: even with noisy proxies and sub-optimal stratification, SIS can improve estimator efficiency compared to IS or SRS individually, though extreme proposal mismatch may limit these gains. Experiments across binary and multiclass tasks demonstrate consistent efficiency improvements under fixed label budgets, underscoring SIS as a principled, label-efficient, and operationally lightweight methodology for post-deployment model monitoring.