Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters

TL;DR

This paper addresses the instability in DARTS neural architecture search by proposing an amended gradient estimation method that improves stability and allows exploration of larger search spaces.

Contribution

It introduces a mathematically guaranteed bounded error gradient estimation technique to enhance DARTS stability and unify hyper-parameter settings across search and re-training stages.

Findings

Significantly improved search stability on CIFAR10 and ImageNet.

Enables exploration of larger, previously uninvestigated search spaces.

Reduces the optimization gap between super-network and sub-networks.

Abstract

DARTS is a popular algorithm for neural architecture search (NAS). Despite its great advantage in search efficiency, DARTS often suffers weak stability, which reflects in the large variation among individual trials as well as the sensitivity to the hyper-parameters of the search process. This paper owes such instability to an optimization gap between the super-network and its sub-networks, namely, improving the validation accuracy of the super-network does not necessarily lead to a higher expectation on the performance of the sampled sub-networks. Then, we point out that the gap is due to the inaccurate estimation of the architectural gradients, based on which we propose an amended estimation method. Mathematically, our method guarantees a bounded error from the true gradients while the original estimation does not. Our approach bridges the gap from two aspects, namely, amending the estimation on the architectural gradients, and unifying the hyper-parameter settings in the search and re-training stages. Experiments on CIFAR10 and ImageNet demonstrate that our approach largely improves search stability and, more importantly, enables DARTS-based approaches to explore much larger search spaces that have not been investigated before.