Towards Understanding the Behaviors of Optimal Deep Active Learning Algorithms

TL;DR

This paper introduces a simulated annealing approach to identify the optimal active learning algorithm, providing insights into its behaviors and improving existing heuristics to advance active learning research.

Contribution

It presents a novel method to find the optimal active learning algorithm and analyzes its behaviors, offering new insights and improvements over existing heuristics.

Findings

The optimal AL oracle exhibits distinct qualitative behaviors.

Insights from the oracle can improve heuristic AL algorithms.

The approach provides a benchmark for evaluating AL strategies.

Abstract

Active learning (AL) algorithms may achieve better performance with fewer data because the model guides the data selection process. While many algorithms have been proposed, there is little study on what the optimal AL algorithm looks like, which would help researchers understand where their models fall short and iterate on the design. In this paper, we present a simulated annealing algorithm to search for this optimal oracle and analyze it for several tasks. We present qualitative and quantitative insights into the behaviors of this oracle, comparing and contrasting them with those of various heuristics. Moreover, we are able to consistently improve the heuristics using one particular insight. We hope that our findings can better inform future active learning research. The code is available at https://github.com/YilunZhou/optimal-active-learning.