Less is More: Proxy Datasets in NAS approaches

TL;DR

This paper investigates the use of smaller proxy datasets in Neural Architecture Search (NAS) to significantly reduce search time while maintaining or improving accuracy, demonstrating that only 25% of data can suffice.

Contribution

It introduces and evaluates dataset sampling techniques as an agnostic method to accelerate NAS, showing substantial efficiency gains without sacrificing performance.

Findings

Reducing dataset size to 25% maintains accuracy

Search time decreases to 25% with smaller datasets

Subset-derived designs can outperform full dataset designs by up to 22 percentage points

Abstract

Neural Architecture Search (NAS) defines the design of Neural Networks as a search problem. Unfortunately, NAS is computationally intensive because of various possibilities depending on the number of elements in the design and the possible connections between them. In this work, we extensively analyze the role of the dataset size based on several sampling approaches for reducing the dataset size (unsupervised and supervised cases) as an agnostic approach to reduce search time. We compared these techniques with four common NAS approaches in NAS-Bench-201 in roughly 1,400 experiments on CIFAR-100. One of our surprising findings is that in most cases we can reduce the amount of training data to 25\%, consequently reducing search time to 25\%, while at the same time maintaining the same accuracy as if training on the full dataset. Additionally, some designs derived from subsets out-perform designs derived from the full dataset by up to 22 p.p. accuracy.