Overcoming Overconfidence for Active Learning

TL;DR

This paper introduces two novel methods, CMaM and RankedMS, to mitigate overconfidence in active learning, improving data selection efficiency under limited labeling budgets.

Contribution

It proposes the first augmentation and selection strategies specifically designed to address overconfidence in active learning scenarios.

Findings

CMaM improves model calibration by expanding training distribution.

RankedMS prevents overly confident data selection.

Methods enhance active learning efficiency with limited data.

Abstract

It is not an exaggeration to say that the recent progress in artificial intelligence technology depends on large-scale and high-quality data. Simultaneously, a prevalent issue exists everywhere: the budget for data labeling is constrained. Active learning is a prominent approach for addressing this issue, where valuable data for labeling is selected through a model and utilized to iteratively adjust the model. However, due to the limited amount of data in each iteration, the model is vulnerable to bias; thus, it is more likely to yield overconfident predictions. In this paper, we present two novel methods to address the problem of overconfidence that arises in the active learning scenario. The first is an augmentation strategy named Cross-Mix-and-Mix (CMaM), which aims to calibrate the model by expanding the limited training distribution. The second is a selection strategy named Ranked Margin Sampling (RankedMS), which prevents choosing data that leads to overly confident predictions. Through various experiments and analyses, we are able to demonstrate that our proposals facilitate efficient data selection by alleviating overconfidence, even though they are readily applicable.