ALPBench: A Benchmark for Active Learning Pipelines on Tabular Data

TL;DR

ALPBench is a comprehensive benchmark suite designed to evaluate and compare active learning pipelines on tabular data, enabling standardized, reproducible assessments of various query strategies and learning algorithms.

Contribution

The paper introduces ALPBench, a standardized benchmark with extensive datasets and settings for evaluating active learning pipelines, addressing the lack of such benchmarks in the field.

Findings

Demonstrated broad compatibility of ALPBench with multiple algorithms and strategies.

Provided insights into the performance variations across different active learning configurations.

Enabled reproducible evaluation of active learning methods on real-world datasets.

Abstract

In settings where only a budgeted amount of labeled data can be afforded, active learning seeks to devise query strategies for selecting the most informative data points to be labeled, aiming to enhance learning algorithms' efficiency and performance. Numerous such query strategies have been proposed and compared in the active learning literature. However, the community still lacks standardized benchmarks for comparing the performance of different query strategies. This particularly holds for the combination of query strategies with different learning algorithms into active learning pipelines and examining the impact of the learning algorithm choice. To close this gap, we propose ALPBench, which facilitates the specification, execution, and performance monitoring of active learning pipelines. It has built-in measures to ensure evaluations are done reproducibly, saving exact dataset splits and hyperparameter settings of used algorithms. In total, ALPBench consists of 86 real-world tabular classification datasets and 5 active learning settings, yielding 430 active learning problems. To demonstrate its usefulness and broad compatibility with various learning algorithms and query strategies, we conduct an exemplary study evaluating 9 query strategies paired with 8 learning algorithms in 2 different settings. We provide ALPBench here: https://github.com/ValentinMargraf/ActiveLearningPipelines.