AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment

TL;DR

AdvantageNAS introduces a gradient-based neural architecture search method that reduces search time by incorporating credit assignment, demonstrating higher performance and theoretical convergence guarantees on benchmark datasets.

Contribution

It proposes a novel, efficient search strategy combining credit assignment with gradient-based NAS, improving search speed and reliability.

Findings

Achieves higher performance architectures within limited time.

Demonstrates theoretical convergence and monotonic expected loss improvement.

Outperforms existing sparse propagation NAS methods on benchmarks.

Abstract

Neural architecture search (NAS) is an approach for automatically designing a neural network architecture without human effort or expert knowledge. However, the high computational cost of NAS limits its use in commercial applications. Two recent NAS paradigms, namely one-shot and sparse propagation, which reduce the time and space complexities, respectively, provide clues for solving this problem. In this paper, we propose a novel search strategy for one-shot and sparse propagation NAS, namely AdvantageNAS, which further reduces the time complexity of NAS by reducing the number of search iterations. AdvantageNAS is a gradient-based approach that improves the search efficiency by introducing credit assignment in gradient estimation for architecture updates. Experiments on the NAS-Bench-201 and PTB dataset show that AdvantageNAS discovers an architecture with higher performance under a limited time budget compared to existing sparse propagation NAS. To further reveal the reliabilities of AdvantageNAS, we investigate it theoretically and find that it monotonically improves the expected loss and thus converges.