A Cross-Domain Benchmark for Active Learning

TL;DR

This paper introduces CDALBench, a comprehensive cross-domain active learning benchmark with extensive repetitions, revealing that method performance varies significantly across domains and runs, emphasizing the need for robust evaluation.

Contribution

It presents the first cross-domain active learning benchmark with high-repetition evaluation, highlighting the importance of domain diversity and multiple runs for reliable AL method assessment.

Findings

Method performance varies across different domains.

High number of repetitions is crucial for reliable evaluation.

Performance of established methods can vary dramatically depending on the seed.

Abstract

Active Learning (AL) deals with identifying the most informative samples for labeling to reduce data annotation costs for supervised learning tasks. AL research suffers from the fact that lifts from literature generalize poorly and that only a small number of repetitions of experiments are conducted. To overcome these obstacles, we propose CDALBench, the first active learning benchmark which includes tasks in computer vision, natural language processing and tabular learning. Furthermore, by providing an efficient, greedy oracle, CDALBench can be evaluated with 50 runs for each experiment. We show, that both the cross-domain character and a large amount of repetitions are crucial for sophisticated evaluation of AL research. Concretely, we show that the superiority of specific methods varies over the different domains, making it important to evaluate Active Learning with a cross-domain benchmark. Additionally, we show that having a large amount of runs is crucial. With only conducting three runs as often done in the literature, the superiority of specific methods can strongly vary with the specific runs. This effect is so strong, that, depending on the seed, even a well-established method's performance can be significantly better and significantly worse than random for the same dataset.