OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strength

TL;DR

This paper introduces OStr-DARTS, a novel operation strength-based criterion for neural architecture search that effectively addresses the degeneration issue in differentiable NAS without altering supernet optimization.

Contribution

It proposes a new operation importance criterion based on operation strength, improving DARTS stability and performance.

Findings

Addresses degeneration issue in DARTS

Operation strength criterion improves architecture quality

Effective on NAS-Bench-201 and DARTS spaces

Abstract

Differentiable architecture search (DARTS) has emerged as a promising technique for effective neural architecture search, and it mainly contains two steps to find the high-performance architecture: First, the DARTS supernet that consists of mixed operations will be optimized via gradient descent. Second, the final architecture will be built by the selected operations that contribute the most to the supernet. Although DARTS improves the efficiency of NAS, it suffers from the well-known degeneration issue which can lead to deteriorating architectures. Existing works mainly attribute the degeneration issue to the failure of its supernet optimization, while little attention has been paid to the selection method. In this paper, we cease to apply the widely-used magnitude-based selection method and propose a novel criterion based on operation strength that estimates the importance of an operation by its effect on the final loss. We show that the degeneration issue can be effectively addressed by using the proposed criterion without any modification of supernet optimization, indicating that the magnitude-based selection method can be a critical reason for the instability of DARTS. The experiments on NAS-Bench-201 and DARTS search spaces show the effectiveness of our method.