Benchmarking of Query Strategies: Towards Future Deep Active Learning

TL;DR

This paper establishes standardized experimental settings for deep active learning (DAL) and evaluates various query strategies across six diverse datasets, including medical and inspection images, to improve benchmarking and understanding of DAL effectiveness.

Contribution

It introduces standardized experimental protocols for DAL and systematically compares query strategies on multiple datasets, including real-world medical and inspection images.

Findings

Standardized settings enable fair comparison of DAL methods.

Certain query strategies outperform others across datasets.

Model-based approaches are effective for diverse data types.

Abstract

In this study, we benchmark query strategies for deep actice learning~(DAL). DAL reduces annotation costs by annotating only high-quality samples selected by query strategies. Existing research has two main problems, that the experimental settings are not standardized, making the evaluation of existing methods is difficult, and that most of experiments were conducted on the CIFAR or MNIST datasets. Therefore, we develop standardized experimental settings for DAL and investigate the effectiveness of various query strategies using six datasets, including those that contain medical and visual inspection images. In addition, since most current DAL approaches are model-based, we perform verification experiments using fully-trained models for querying to investigate the effectiveness of these approaches for the six datasets. Our code is available at \href{https://github.com/ia-gu/Benchmarking-of-Query-Strategies-Towards-Future-Deep-Active-Learning}