Half Search Space is All You Need

TL;DR

This paper introduces a method combining Zero-Shot NAS and One-Shot NAS to significantly reduce GPU memory usage during neural architecture search without sacrificing accuracy.

Contribution

It proposes an automatic search space pruning technique using Zero-Shot NAS to improve efficiency of One-Shot NAS methods.

Findings

81% reduction in memory consumption

Maintains same accuracy as baseline

Effective pruning of low-performing architectures

Abstract

Neural Architecture Search (NAS) is a powerful tool for automating architecture design. One-Shot NAS techniques, such as DARTS, have gained substantial popularity due to their combination of search efficiency with simplicity of implementation. By design, One-Shot methods have high GPU memory requirements during the search. To mitigate this issue, we propose to prune the search space in an efficient automatic manner to reduce memory consumption and search time while preserving the search accuracy. Specifically, we utilise Zero-Shot NAS to efficiently remove low-performing architectures from the search space before applying One-Shot NAS to the pruned search space. Experimental results on the DARTS search space show that our approach reduces memory consumption by 81% compared to the baseline One-Shot setup while achieving the same level of accuracy.