Zero-Cost Operation Scoring in Differentiable Architecture Search
Lichuan Xiang, {\L}ukasz Dudziak, Mohamed S. Abdelfattah, Thomas Chau,, Nicholas D. Lane, Hongkai Wen

TL;DR
This paper introduces a novel zero-cost operation scoring method for differentiable neural architecture search, significantly improving search efficiency and accuracy across multiple benchmarks and large search spaces.
Contribution
It proposes a perturbation-based zero-cost scoring approach that outperforms existing proxies in NAS, enabling faster and more accurate architecture search.
Findings
Significant accuracy improvements on NAS benchmarks.
Over 10x faster search times on ImageNet with DARTS space.
Better transferability of searched architectures.
Abstract
We formalize and analyze a fundamental component of differentiable neural architecture search (NAS): local "operation scoring" at each operation choice. We view existing operation scoring functions as inexact proxies for accuracy, and we find that they perform poorly when analyzed empirically on NAS benchmarks. From this perspective, we introduce a novel \textit{perturbation-based zero-cost operation scoring} (Zero-Cost-PT) approach, which utilizes zero-cost proxies that were recently studied in multi-trial NAS but degrade significantly on larger search spaces, typical for differentiable NAS. We conduct a thorough empirical evaluation on a number of NAS benchmarks and large search spaces, from NAS-Bench-201, NAS-Bench-1Shot1, NAS-Bench-Macro, to DARTS-like and MobileNet-like spaces, showing significant improvements in both search time and accuracy. On the ImageNet classification task on…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsAdvanced Neural Network Applications · Adversarial Robustness in Machine Learning · Machine Learning in Materials Science
MethodsDifferentiable Architecture Search · Differentiable Neural Architecture Search
