DARTS+: Improved Differentiable Architecture Search with Early Stopping

TL;DR

DARTS+ introduces an early stopping criterion to improve differentiable architecture search, preventing overfitting and collapse in search performance, leading to better results on benchmark datasets.

Contribution

The paper proposes a simple early stopping method for DARTS that effectively prevents performance collapse and overfitting during architecture search.

Findings

DARTS+ achieves 2.32% test error on CIFAR10.

DARTS+ achieves 14.87% test error on CIFAR100.

DARTS+ achieves 23.7% test error on ImageNet.

Abstract

Recently, there has been a growing interest in automating the process of neural architecture design, and the Differentiable Architecture Search (DARTS) method makes the process available within a few GPU days. However, the performance of DARTS is often observed to collapse when the number of search epochs becomes large. Meanwhile, lots of "{\em skip-connect}s" are found in the selected architectures. In this paper, we claim that the cause of the collapse is that there exists overfitting in the optimization of DARTS. Therefore, we propose a simple and effective algorithm, named "DARTS+", to avoid the collapse and improve the original DARTS, by "early stopping" the search procedure when meeting a certain criterion. We also conduct comprehensive experiments on benchmark datasets and different search spaces and show the effectiveness of our DARTS+ algorithm, and DARTS+ achieves $2.32\%$ test error on CIFAR10, $14.87\%$ on CIFAR100, and $23.7\%$ on ImageNet. We further remark that the idea of "early stopping" is implicitly included in some existing DARTS variants by manually setting a small number of search epochs, while we give an {\em explicit} criterion for "early stopping".