Neural Architecture Search: Two Constant Shared Weights Initialisations

TL;DR

This paper introduces epsinas, a fast, zero-cost neural architecture evaluation metric based on two constant shared weight initialisations, which correlates strongly with trained accuracy across various tasks and datasets.

Contribution

The paper proposes epsinas, a novel zero-cost NAS metric using constant shared weight initialisations and output statistics, requiring no training or labels, and enabling rapid architecture evaluation.

Findings

Strong correlation with trained accuracy across tasks

Operates in a fraction of a GPU second

No need for data labels or gradient computation

Abstract

In the last decade, zero-cost metrics have gained prominence in neural architecture search (NAS) due to their ability to evaluate architectures without training. These metrics are significantly faster and less computationally expensive than traditional NAS methods and provide insights into neural architectures' internal workings. This paper introduces epsinas, a novel zero-cost NAS metric that assesses architecture potential using two constant shared weight initialisations and the statistics of their outputs. We show that the dispersion of raw outputs, normalised by their average magnitude, strongly correlates with trained accuracy. This effect holds across image classification and language tasks on NAS-Bench-101, NAS-Bench-201, and NAS-Bench-NLP. Our method requires no data labels, operates on a single minibatch, and eliminates the need for gradient computation, making it independent of training hyperparameters, loss metrics, and human annotations. It evaluates a network in a fraction of a GPU second and integrates seamlessly into existing NAS frameworks. The code supporting this study can be found on GitHub at https://github.com/egracheva/epsinas.