ASP: Automatic Selection of Proxy dataset for efficient AutoML

TL;DR

This paper introduces ASP, a dynamic method for selecting informative proxy datasets during training, significantly reducing AutoML computation time while improving model performance across multiple benchmarks.

Contribution

ASP is a novel framework that adaptively selects proxy datasets at each epoch, enhancing AutoML efficiency and effectiveness compared to existing data selection methods.

Findings

ASP achieves 2x-20x speedup in AutoML processing.

ASP outperforms other data selection methods at all ratios.

ASP improves model accuracy on multiple benchmarks.

Abstract

Deep neural networks have gained great success due to the increasing amounts of data, and diverse effective neural network designs. However, it also brings a heavy computing burden as the amount of training data is proportional to the training time. In addition, a well-behaved model requires repeated trials of different structure designs and hyper-parameters, which may take a large amount of time even with state-of-the-art (SOTA) hyper-parameter optimization (HPO) algorithms and neural architecture search (NAS) algorithms. In this paper, we propose an Automatic Selection of Proxy dataset framework (ASP) aimed to dynamically find the informative proxy subsets of training data at each epoch, reducing the training data size as well as saving the AutoML processing time. We verify the effectiveness and generalization of ASP on CIFAR10, CIFAR100, ImageNet16-120, and ImageNet-1k, across various public model benchmarks. The experiment results show that ASP can obtain better results than other data selection methods at all selection ratios. ASP can also enable much more efficient AutoML processing with a speedup of 2x-20x while obtaining better architectures and better hyper-parameters compared to utilizing the entire dataset.