Pearls from Pebbles: Improved Confidence Functions for Auto-labeling

TL;DR

This paper introduces olander, a novel confidence function designed to improve threshold-based auto-labeling by maximizing coverage and reliability, outperforming calibration methods significantly.

Contribution

The paper proposes olander, a new framework and post-hoc method for optimal confidence functions in auto-labeling, addressing overconfidence issues and enhancing performance.

Findings

olander achieves up to 60% higher coverage than baselines.

It maintains auto-labeling error below 5%.

Uses the same amount of labeled data as existing methods.

Abstract

Auto-labeling is an important family of techniques that produce labeled training sets with minimum manual labeling. A prominent variant, threshold-based auto-labeling (TBAL), works by finding a threshold on a model's confidence scores above which it can accurately label unlabeled data points. However, many models are known to produce overconfident scores, leading to poor TBAL performance. While a natural idea is to apply off-the-shelf calibration methods to alleviate the overconfidence issue, such methods still fall short. Rather than experimenting with ad-hoc choices of confidence functions, we propose a framework for studying the \emph{optimal} TBAL confidence function. We develop a tractable version of the framework to obtain \texttt{Colander} (Confidence functions for Efficient and Reliable Auto-labeling), a new post-hoc method specifically designed to maximize performance in TBAL systems. We perform an extensive empirical evaluation of our method \texttt{Colander} and compare it against methods designed for calibration. \texttt{Colander} achieves up to 60\% improvements on coverage over the baselines while maintaining auto-labeling error below $5\%$ and using the same amount of labeled data as the baselines.