# Training Data Subset Search with Ensemble Active Learning

## TL;DR

This paper introduces a scalable ensemble active learning method to identify optimal training data subsets, improving deep neural network performance and efficiency on large-scale image classification and object detection tasks.

## Contribution

It extends ensemble active learning to large datasets using checkpoints, enabling efficient subset search that enhances model accuracy and training efficiency.

## Key findings

- Favorable training data subsets improve DNN accuracy.
- Ensemble active learning scales to 500k samples efficiently.
- Significant benefits in large-scale vision tasks.

## Abstract

Deep Neural Networks (DNNs) often rely on very large datasets for training. Given the large size of such datasets, it is conceivable that they contain certain samples that either do not contribute or negatively impact the DNN's optimization. Modifying the training distribution in a way that excludes such samples could provide an effective solution to both improve performance and reduce training time. In this paper, we propose to scale up ensemble Active Learning (AL) methods to perform acquisition at a large scale (10k to 500k samples at a time). We do this with ensembles of hundreds of models, obtained at a minimal computational cost by reusing intermediate training checkpoints. This allows us to automatically and efficiently perform a training data subset search for large labeled datasets. We observe that our approach obtains favorable subsets of training data, which can be used to train more accurate DNNs than training with the entire dataset. We perform an extensive experimental study of this phenomenon on three image classification benchmarks (CIFAR-10, CIFAR-100 and ImageNet), as well as an internal object detection benchmark for prototyping perception models for autonomous driving. Unlike existing studies, our experiments on object detection are at the scale required for production-ready autonomous driving systems. We provide insights on the impact of different initialization schemes, acquisition functions and ensemble configurations at this scale. Our results provide strong empirical evidence that optimizing the training data distribution can provide significant benefits on large scale vision tasks.