Single-DARTS: Towards Stable Architecture Search

TL;DR

Single-DARTS introduces a single-level optimization approach to neural architecture search, significantly improving stability and performance over traditional bi-level methods by addressing the issue of performance collapse caused by dominating operations.

Contribution

It is the first to systematically analyze and implement single-level optimization in DARTS, alleviating performance collapse and enhancing search stability.

Findings

Single-DARTS achieves state-of-the-art results on NAS-Benchmark-201.

It nearly finds optimal architectures in the benchmark.

Single-level optimization is more stable than bi-level optimization.

Abstract

Differentiable architecture search (DARTS) marks a milestone in Neural Architecture Search (NAS), boasting simplicity and small search costs. However, DARTS still suffers from frequent performance collapse, which happens when some operations, such as skip connections, zeroes and poolings, dominate the architecture. In this paper, we are the first to point out that the phenomenon is attributed to bi-level optimization. We propose Single-DARTS which merely uses single-level optimization, updating network weights and architecture parameters simultaneously with the same data batch. Even single-level optimization has been previously attempted, no literature provides a systematic explanation on this essential point. Replacing the bi-level optimization, Single-DARTS obviously alleviates performance collapse as well as enhances the stability of architecture search. Experiment results show that Single-DARTS achieves state-of-the-art performance on mainstream search spaces. For instance, on NAS-Benchmark-201, the searched architectures are nearly optimal ones. We also validate that the single-level optimization framework is much more stable than the bi-level one. We hope that this simple yet effective method will give some insights on differential architecture search. The code is available at https://github.com/PencilAndBike/Single-DARTS.git.