MS-DARTS: Mean-Shift Based Differentiable Architecture Search

TL;DR

MS-DARTS introduces a mean-shift technique to enhance the stability and accuracy of differentiable architecture search, effectively smoothing the loss landscape and reducing performance deterioration during discretization.

Contribution

The paper proposes a novel mean-shift based method to improve stability and performance in DARTS, addressing discretization issues in NAS.

Findings

MS-DARTS achieves higher accuracy on CIFAR-10, CIFAR-100, and ImageNet.

The method reduces search cost compared to state-of-the-art NAS techniques.

Improves stability and convergence of architecture search.

Abstract

Differentiable Architecture Search (DARTS) is an effective continuous relaxation-based network architecture search (NAS) method with low search cost. It has attracted significant attentions in Auto-ML research and becomes one of the most useful paradigms in NAS. Although DARTS can produce superior efficiency over traditional NAS approaches with better control of complex parameters, oftentimes it suffers from stabilization issues in producing deteriorating architectures when discretizing the continuous architecture. We observed considerable loss of validity causing dramatic decline in performance at this final discretization step of DARTS. To address this issue, we propose a Mean-Shift based DARTS (MS-DARTS) to improve stability based on sampling and perturbation. Our approach can improve bot the stability and accuracy of DARTS, by smoothing the loss landscape and sampling architecture parameters within a suitable bandwidth. We investigate the convergence of our mean-shift approach, together with the effects of bandwidth selection that affects stability and accuracy. Evaluations performed on CIFAR-10, CIFAR-100, and ImageNet show that MS-DARTS archives higher performance over other state-of-the-art NAS methods with reduced search cost.