Rebuilding Trust in Active Learning with Actionable Metrics

TL;DR

This paper introduces actionable metrics for active learning to improve interpretability and trust among industry practitioners, demonstrating their effectiveness through extensive experiments on multiple datasets.

Contribution

It proposes new actionable metrics that enhance interpretability of active learning strategies, addressing industry concerns about performance guarantees and consistency.

Findings

Metrics improve interpretability of AL strategies

Experiments on CIFAR100, Fashion-MNIST, 20Newsgroups validate effectiveness

Metrics help identify causes of poor AL performance

Abstract

Active Learning (AL) is an active domain of research, but is seldom used in the industry despite the pressing needs. This is in part due to a misalignment of objectives, while research strives at getting the best results on selected datasets, the industry wants guarantees that Active Learning will perform consistently and at least better than random labeling. The very one-off nature of Active Learning makes it crucial to understand how strategy selection can be carried out and what drives poor performance (lack of exploration, selection of samples that are too hard to classify, ...). To help rebuild trust of industrial practitioners in Active Learning, we present various actionable metrics. Through extensive experiments on reference datasets such as CIFAR100, Fashion-MNIST, and 20Newsgroups, we show that those metrics brings interpretability to AL strategies that can be leveraged by the practitioner.