Reducing Neural Architecture Search Spaces with Training-Free Statistics and Computational Graph Clustering

TL;DR

This paper introduces C-BRED, a clustering-based method that reduces NAS search spaces by selecting promising architecture clusters using training-free proxy statistics, thereby improving efficiency and accuracy.

Contribution

C-BRED is a novel clustering technique that leverages training-free statistics to effectively reduce NAS search spaces while maintaining high accuracy.

Findings

C-BRED achieves 70% accuracy with a reduced search space.

C-BRED outperforms baseline methods in NAS space reduction.

The method significantly lowers computational costs of NAS.

Abstract

The computational demands of neural architecture search (NAS) algorithms are usually directly proportional to the size of their target search spaces. Thus, limiting the search to high-quality subsets can greatly reduce the computational load of NAS algorithms. In this paper, we present Clustering-Based REDuction (C-BRED), a new technique to reduce the size of NAS search spaces. C-BRED reduces a NAS space by clustering the computational graphs associated with its architectures and selecting the most promising cluster using proxy statistics correlated with network accuracy. When considering the NAS-Bench-201 (NB201) data set and the CIFAR-100 task, C-BRED selects a subset with 70% average accuracy instead of the whole space's 64% average accuracy.