An Analysis of Super-Net Heuristics in Weight-Sharing NAS
Kaicheng Yu, Ren\'e Ranftl, Mathieu Salzmann

TL;DR
This paper systematically studies the impact of various training heuristics on super-net performance in weight-sharing NAS, revealing that proper hyper-parameter tuning is crucial and simple methods can rival complex algorithms.
Contribution
It disentangles super-net training from search algorithms, evaluates 14 heuristics across benchmarks, and highlights the importance of hyper-parameter tuning for effective NAS.
Findings
Certain heuristics negatively affect super-net performance.
Proper hyper-parameter settings are key to strong results.
Simple random search can match complex NAS methods when well-trained.
Abstract
Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design and train the shared-weight backbone network, a.k.a. the super-net. Since heuristics substantially vary across different methods and have not been carefully studied, it is unclear to which extent they impact super-net training and hence the weight-sharing NAS algorithms. In this paper, we disentangle super-net training from the search algorithm, isolate 14 frequently-used training heuristics, and evaluate them over three benchmark search spaces. Our analysis uncovers that several commonly-used heuristics negatively impact the correlation between super-net and stand-alone performance, whereas simple, but often overlooked factors, such as proper hyper-parameter settings, are key to achieve strong performance.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Machine Learning and ELM · Machine Learning and Data Classification
MethodsRandom Search
