Half Search Space is All You Need
Pavel Rumiantsev, Mark Coates

TL;DR
This paper introduces a method combining Zero-Shot NAS and One-Shot NAS to significantly reduce GPU memory usage during neural architecture search without sacrificing accuracy.
Contribution
It proposes an automatic search space pruning technique using Zero-Shot NAS to improve efficiency of One-Shot NAS methods.
Findings
81% reduction in memory consumption
Maintains same accuracy as baseline
Effective pruning of low-performing architectures
Abstract
Neural Architecture Search (NAS) is a powerful tool for automating architecture design. One-Shot NAS techniques, such as DARTS, have gained substantial popularity due to their combination of search efficiency with simplicity of implementation. By design, One-Shot methods have high GPU memory requirements during the search. To mitigate this issue, we propose to prune the search space in an efficient automatic manner to reduce memory consumption and search time while preserving the search accuracy. Specifically, we utilise Zero-Shot NAS to efficiently remove low-performing architectures from the search space before applying One-Shot NAS to the pruned search space. Experimental results on the DARTS search space show that our approach reduces memory consumption by 81% compared to the baseline One-Shot setup while achieving the same level of accuracy.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Machine Learning and Data Classification · Adversarial Robustness in Machine Learning
MethodsDifferentiable Architecture Search
