An Analysis of Super-Net Heuristics in Weight-Sharing NAS

TL;DR

This paper systematically studies the impact of various training heuristics on super-net performance in weight-sharing NAS, revealing that proper hyper-parameter tuning is crucial and simple methods can rival complex algorithms.

Contribution

It disentangles super-net training from search algorithms, evaluates 14 heuristics across benchmarks, and highlights the importance of hyper-parameter tuning for effective NAS.

Findings

Certain heuristics negatively affect super-net performance.

Proper hyper-parameter settings are key to strong results.

Simple random search can match complex NAS methods when well-trained.

Abstract

Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design and train the shared-weight backbone network, a.k.a. the super-net. Since heuristics substantially vary across different methods and have not been carefully studied, it is unclear to which extent they impact super-net training and hence the weight-sharing NAS algorithms. In this paper, we disentangle super-net training from the search algorithm, isolate 14 frequently-used training heuristics, and evaluate them over three benchmark search spaces. Our analysis uncovers that several commonly-used heuristics negatively impact the correlation between super-net and stand-alone performance, whereas simple, but often overlooked factors, such as proper hyper-parameter settings, are key to achieve strong performance. Equipped with this knowledge, we show that simple random search achieves competitive performance to complex state-of-the-art NAS algorithms when the super-net is properly trained.