LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learning

TL;DR

This paper introduces LabelBench, a comprehensive and efficient framework for evaluating combined label-efficient learning techniques, and demonstrates its effectiveness with a new benchmark on vision transformers.

Contribution

It presents LabelBench, a novel modular framework for joint evaluation of multiple label-efficient learning methods, and provides a new benchmark showing improved label-efficiency in active learning with semi-supervised learning.

Findings

LabelBench enables comprehensive evaluation of label-efficient methods.

The benchmark shows improved label-efficiency over previous methods.

Open-source code facilitates community contributions.

Abstract

Labeled data are critical to modern machine learning applications, but obtaining labels can be expensive. To mitigate this cost, machine learning methods, such as transfer learning, semi-supervised learning and active learning, aim to be label-efficient: achieving high predictive performance from relatively few labeled examples. While obtaining the best label-efficiency in practice often requires combinations of these techniques, existing benchmark and evaluation frameworks do not capture a concerted combination of all such techniques. This paper addresses this deficiency by introducing LabelBench, a new computationally-efficient framework for joint evaluation of multiple label-efficient learning techniques. As an application of LabelBench, we introduce a novel benchmark of state-of-the-art active learning methods in combination with semi-supervised learning for fine-tuning pretrained vision transformers. Our benchmark demonstrates better label-efficiencies than previously reported in active learning. LabelBench's modular codebase is open-sourced for the broader community to contribute label-efficient learning methods and benchmarks. The repository can be found at: https://github.com/EfficientTraining/LabelBench.