When does Active Learning Work?

TL;DR

This paper conducts a comprehensive experimental study to evaluate when and how effectively Active Learning improves classifier performance across diverse tasks and settings.

Contribution

It introduces a detailed methodology for assessing Active Learning performance and explores its effectiveness across various classifiers and tasks.

Findings

Active Learning's effectiveness varies by task and classifier.

A new methodology for performance assessment is proposed.

Results identify conditions where AL provides significant benefits.

Abstract

Active Learning (AL) methods seek to improve classifier performance when labels are expensive or scarce. We consider two central questions: Where does AL work? How much does it help? To address these questions, a comprehensive experimental simulation study of Active Learning is presented. We consider a variety of tasks, classifiers and other AL factors, to present a broad exploration of AL performance in various settings. A precise way to quantify performance is needed in order to know when AL works. Thus we also present a detailed methodology for tackling the complexities of assessing AL performance in the context of this experimental study.