ActiveLab: Active Learning with Re-Labeling by Multiple Annotators

TL;DR

ActiveLab is a versatile active learning method that optimally balances re-labeling and new labeling to improve classifier accuracy efficiently with limited annotations.

Contribution

It introduces a practical, model-agnostic approach for active learning that automatically decides when to re-label or label new data, enhancing annotation efficiency.

Findings

ActiveLab outperforms existing active learning methods in accuracy and annotation efficiency.

It effectively estimates when re-labeling is more informative than new labeling.

Demonstrated success on image and tabular datasets.

Abstract

In real-world data labeling applications, annotators often provide imperfect labels. It is thus common to employ multiple annotators to label data with some overlap between their examples. We study active learning in such settings, aiming to train an accurate classifier by collecting a dataset with the fewest total annotations. Here we propose ActiveLab, a practical method to decide what to label next that works with any classifier model and can be used in pool-based batch active learning with one or multiple annotators. ActiveLab automatically estimates when it is more informative to re-label examples vs. labeling entirely new ones. This is a key aspect of producing high quality labels and trained models within a limited annotation budget. In experiments on image and tabular data, ActiveLab reliably trains more accurate classifiers with far fewer annotations than a wide variety of popular active learning methods.