Learning active learning at the crossroads? evaluation and discussion

TL;DR

This paper evaluates meta-learning strategies for active learning against traditional margin sampling, highlighting their comparative performance across multiple datasets and discussing future research directions.

Contribution

It provides a comprehensive benchmark comparing meta-learned active learning strategies with margin sampling on 20 datasets, offering insights into their relative effectiveness.

Findings

Meta-learning strategies show competitive performance with margin sampling.

No single strategy outperforms others across all datasets.

Lessons learned and future research directions are discussed.

Abstract

Active learning aims to reduce annotation cost by predicting which samples are useful for a human expert to label. Although this field is quite old, several important challenges to using active learning in real-world settings still remain unsolved. In particular, most selection strategies are hand-designed, and it has become clear that there is no best active learning strategy that consistently outperforms all others in all applications. This has motivated research into meta-learning algorithms for "learning how to actively learn". In this paper, we compare this kind of approach with the association of a Random Forest with the margin sampling strategy, reported in recent comparative studies as a very competitive heuristic. To this end, we present the results of a benchmark performed on 20 datasets that compares a strategy learned using a recent meta-learning algorithm with margin sampling. We also present some lessons learned and open future perspectives.