ZARTS: On Zero-order Optimization for Neural Architecture Search

TL;DR

ZARTS introduces a zero-order optimization approach for neural architecture search, avoiding the biased gradients of differentiable methods like DARTS, leading to more robust and accurate architectures.

Contribution

The paper proposes ZARTS, a zero-order NAS method that outperforms DARTS in robustness and accuracy by eliminating the need for gradient-based approximations.

Findings

ZARTS achieves state-of-the-art accuracy on CIFAR-10 and ImageNet.

ZARTS demonstrates superior robustness over DARTS across multiple benchmarks.

MGS, a zero-order method, balances accuracy and computational efficiency.

Abstract

Differentiable architecture search (DARTS) has been a popular one-shot paradigm for NAS due to its high efficiency. It introduces trainable architecture parameters to represent the importance of candidate operations and proposes first/second-order approximation to estimate their gradients, making it possible to solve NAS by gradient descent algorithm. However, our in-depth empirical results show that the approximation will often distort the loss landscape, leading to the biased objective to optimize and in turn inaccurate gradient estimation for architecture parameters. This work turns to zero-order optimization and proposes a novel NAS scheme, called ZARTS, to search without enforcing the above approximation. Specifically, three representative zero-order optimization methods are introduced: RS, MGS, and GLD, among which MGS performs best by balancing the accuracy and speed. Moreover, we explore the connections between RS/MGS and gradient descent algorithm and show that our ZARTS can be seen as a robust gradient-free counterpart to DARTS. Extensive experiments on multiple datasets and search spaces show the remarkable performance of our method. In particular, results on 12 benchmarks verify the outstanding robustness of ZARTS, where the performance of DARTS collapses due to its known instability issue. Also, we search on the search space of DARTS to compare with peer methods, and our discovered architecture achieves 97.54% accuracy on CIFAR-10 and 75.7% top-1 accuracy on ImageNet, which are state-of-the-art performance.