Understanding Uncertainty-based Active Learning Under Model Mismatch

TL;DR

This paper investigates how the capacity of machine learning models influences the effectiveness of Uncertainty-based Active Learning, revealing that low-capacity models can perform worse than random sampling, and suggesting alternative strategies for such cases.

Contribution

The study provides a theoretical and empirical analysis of UAL under model mismatch, highlighting the impact of model capacity on its performance and proposing improved acquisition strategies.

Findings

UAL can underperform compared to random sampling with low-capacity models

Model capacity significantly affects UAL efficacy

Targeted acquisition functions can enhance UAL performance in low-capacity scenarios

Abstract

Instead of randomly acquiring training data points, Uncertainty-based Active Learning (UAL) operates by querying the label(s) of pivotal samples from an unlabeled pool selected based on the prediction uncertainty, thereby aiming at minimizing the labeling cost for model training. The efficacy of UAL critically depends on the model capacity as well as the adopted uncertainty-based acquisition function. Within the context of this study, our analytical focus is directed toward comprehending how the capacity of the machine learning model may affect UAL efficacy. Through theoretical analysis, comprehensive simulations, and empirical studies, we conclusively demonstrate that UAL can lead to worse performance in comparison with random sampling when the machine learning model class has low capacity and is unable to cover the underlying ground truth. In such situations, adopting acquisition functions that directly target estimating the prediction performance may be beneficial for improving the performance of UAL.