Accelerating Neural Architecture Search via Proxy Data

TL;DR

This paper introduces a novel proxy data selection method for neural architecture search that reduces computational costs significantly while maintaining search quality, enabling faster NAS on large datasets.

Contribution

A new proxy data selection method based on data entropy tailored for NAS, improving efficiency without sacrificing performance.

Findings

Reduces NAS search time to 40 minutes on CIFAR-10 and 7.5 hours on ImageNet.

Achieves state-of-the-art transfer performance with 2.4% error on CIFAR-10.

Proposes a data entropy-based selection method that outperforms existing approaches.

Abstract

Despite the increasing interest in neural architecture search (NAS), the significant computational cost of NAS is a hindrance to researchers. Hence, we propose to reduce the cost of NAS using proxy data, i.e., a representative subset of the target data, without sacrificing search performance. Even though data selection has been used across various fields, our evaluation of existing selection methods for NAS algorithms offered by NAS-Bench-1shot1 reveals that they are not always appropriate for NAS and a new selection method is necessary. By analyzing proxy data constructed using various selection methods through data entropy, we propose a novel proxy data selection method tailored for NAS. To empirically demonstrate the effectiveness, we conduct thorough experiments across diverse datasets, search spaces, and NAS algorithms. Consequently, NAS algorithms with the proposed selection discover architectures that are competitive with those obtained using the entire dataset. It significantly reduces the search cost: executing DARTS with the proposed selection requires only 40 minutes on CIFAR-10 and 7.5 hours on ImageNet with a single GPU. Additionally, when the architecture searched on ImageNet using the proposed selection is inversely transferred to CIFAR-10, a state-of-the-art test error of 2.4\% is yielded. Our code is available at https://github.com/nabk89/NAS-with-Proxy-data.