Differentiable Architecture Search with Random Features

TL;DR

This paper addresses performance collapse in differentiable architecture search (DARTS) by introducing random features, which improve search stability and accuracy, leading to state-of-the-art results on CIFAR-10 and ImageNet datasets.

Contribution

The authors propose a novel approach using random features to mitigate DARTS performance collapse, with theoretical analysis and practical instantiations RF-DARTS and RF-PCDARTS.

Findings

RF-DARTS achieves 94.36% on CIFAR-10.

RF-DARTS attains 24.0% top-1 error on ImageNet.

RF-PCDARTS surpasses existing methods with 23.9% top-1 error on ImageNet.

Abstract

Differentiable architecture search (DARTS) has significantly promoted the development of NAS techniques because of its high search efficiency and effectiveness but suffers from performance collapse. In this paper, we make efforts to alleviate the performance collapse problem for DARTS from two aspects. First, we investigate the expressive power of the supernet in DARTS and then derive a new setup of DARTS paradigm with only training BatchNorm. Second, we theoretically find that random features dilute the auxiliary connection role of skip-connection in supernet optimization and enable search algorithm focus on fairer operation selection, thereby solving the performance collapse problem. We instantiate DARTS and PC-DARTS with random features to build an improved version for each named RF-DARTS and RF-PCDARTS respectively. Experimental results show that RF-DARTS obtains \textbf{94.36\%} test accuracy on CIFAR-10 (which is the nearest optimal result in NAS-Bench-201), and achieves the newest state-of-the-art top-1 test error of \textbf{24.0\%} on ImageNet when transferring from CIFAR-10. Moreover, RF-DARTS performs robustly across three datasets (CIFAR-10, CIFAR-100, and SVHN) and four search spaces (S1-S4). Besides, RF-PCDARTS achieves even better results on ImageNet, that is, \textbf{23.9\%} top-1 and \textbf{7.1\%} top-5 test error, surpassing representative methods like single-path, training-free, and partial-channel paradigms directly searched on ImageNet.