Algorithm Selection for Deep Active Learning with Imbalanced Datasets

TL;DR

This paper introduces TAILOR, an adaptive algorithm that dynamically selects the best active learning strategy for deep models on imbalanced datasets, improving label efficiency and accuracy.

Contribution

The paper presents the first adaptive algorithm selection method, TAILOR, for deep active learning that outperforms or matches the best candidate algorithms across various datasets.

Findings

TAILOR achieves comparable or better accuracy than the best candidate algorithms.

Extensive experiments demonstrate TAILOR's effectiveness in multi-class and multi-label tasks.

The approach improves label efficiency in deep learning applications.

Abstract

Label efficiency has become an increasingly important objective in deep learning applications. Active learning aims to reduce the number of labeled examples needed to train deep networks, but the empirical performance of active learning algorithms can vary dramatically across datasets and applications. It is difficult to know in advance which active learning strategy will perform well or best in a given application. To address this, we propose the first adaptive algorithm selection strategy for deep active learning. For any unlabeled dataset, our (meta) algorithm TAILOR (Thompson ActIve Learning algORithm selection) iteratively and adaptively chooses among a set of candidate active learning algorithms. TAILOR uses novel reward functions aimed at gathering class-balanced examples. Extensive experiments in multi-class and multi-label applications demonstrate TAILOR's effectiveness in achieving accuracy comparable or better than that of the best of the candidate algorithms. Our implementation of TAILOR is open-sourced at https://github.com/jifanz/TAILOR.