Lessons from the Clustering Analysis of a Search Space: A Centroid-based Approach to Initializing NAS

TL;DR

This paper introduces a data-driven, centroid-based initialization method for neural architecture search (NAS) that accelerates convergence and improves final performance by leveraging clustering analysis of the search space.

Contribution

It presents a novel two-step methodology for NAS initialization using clustering analysis and demonstrates its effectiveness on NAS-bench-101.

Findings

Faster convergence compared to random initialization

Improved final performance of NAS algorithms

Effective use of NAS benchmarks for initialization

Abstract

Lots of effort in neural architecture search (NAS) research has been dedicated to algorithmic development, aiming at designing more efficient and less costly methods. Nonetheless, the investigation of the initialization of these techniques remain scare, and currently most NAS methodologies rely on stochastic initialization procedures, because acquiring information prior to search is costly. However, the recent availability of NAS benchmarks have enabled low computational resources prototyping. In this study, we propose to accelerate a NAS algorithm using a data-driven initialization technique, leveraging the availability of NAS benchmarks. Particularly, we proposed a two-step methodology. First, a calibrated clustering analysis of the search space is performed. Second, the centroids are extracted and used to initialize a NAS algorithm. We tested our proposal using Aging Evolution, an evolutionary algorithm, on NAS-bench-101. The results show that, compared to a random initialization, a faster convergence and a better performance of the final solution is achieved.