Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks
Arber Zela, Julien Siems, Lucas Zimmer, Jovita Lukasik, Margret, Keuper, Frank Hutter

TL;DR
This paper introduces surrogate NAS benchmarks that enable efficient and accurate evaluation of neural architecture search methods in large, complex search spaces, overcoming the limitations of traditional tabular benchmarks.
Contribution
It proposes a methodology to create surrogate NAS benchmarks for arbitrary search spaces, including very large ones, improving evaluation fidelity and computational efficiency.
Findings
Surrogate benchmarks outperform tabular benchmarks in modeling true architecture performance.
They provide faithful estimates of NAS method effectiveness on original spaces.
Open-source code facilitates adoption and further research.
Abstract
The most significant barrier to the advancement of Neural Architecture Search (NAS) is its demand for large computational resources, which hinders scientifically sound empirical evaluations of NAS methods. Tabular NAS benchmarks have alleviated this problem substantially, making it possible to properly evaluate NAS methods in seconds on commodity machines. However, an unintended consequence of tabular NAS benchmarks has been a focus on extremely small architectural search spaces since their construction relies on exhaustive evaluations of the space. This leads to unrealistic results that do not transfer to larger spaces. To overcome this fundamental limitation, we propose a methodology to create cheap NAS surrogate benchmarks for arbitrary search spaces. We exemplify this approach by creating surrogate NAS benchmarks on the existing tabular NAS-Bench-101 and on two widely used NAS…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsAdvanced Neural Network Applications · Machine Learning and Data Classification · Machine Learning and Algorithms
MethodsDeep Ensembles · Stochastic Gradient Descent
