A Data-driven Approach to Neural Architecture Search Initialization

TL;DR

This paper introduces a data-driven initialization method for neural architecture search that improves long-term performance and solution quality by using clustering analysis to seed population-based algorithms.

Contribution

It proposes a novel two-step data-driven initialization technique for NAS that enhances search efficiency and solution quality over random and Latin hypercube sampling methods.

Findings

Significant long-term improvements in NAS performance.

Better retrieval of high-fitness local optima.

Effective initialization across various search scenarios.

Abstract

Algorithmic design in neural architecture search (NAS) has received a lot of attention, aiming to improve performance and reduce computational cost. Despite the great advances made, few authors have proposed to tailor initialization techniques for NAS. However, literature shows that a good initial set of solutions facilitate finding the optima. Therefore, in this study, we propose a data-driven technique to initialize a population-based NAS algorithm. Particularly, we proposed a two-step methodology. First, we perform a calibrated clustering analysis of the search space, and second, we extract the centroids and use them to initialize a NAS algorithm. We benchmark our proposed approach against random and Latin hypercube sampling initialization using three population-based algorithms, namely a genetic algorithm, evolutionary algorithm, and aging evolution, on CIFAR-10. More specifically, we use NAS-Bench-101 to leverage the availability of NAS benchmarks. The results show that compared to random and Latin hypercube sampling, the proposed initialization technique enables achieving significant long-term improvements for two of the search baselines, and sometimes in various search scenarios (various training budgets). Moreover, we analyze the distributions of solutions obtained and find that that the population provided by the data-driven initialization technique enables retrieving local optima (maxima) of high fitness and similar configurations.