Stabilizing Differentiable Architecture Search via Perturbation-based Regularization

TL;DR

This paper introduces SDARTS, a regularization method that stabilizes differentiable architecture search by smoothing the loss landscape, leading to better performance and generalizability across multiple datasets and search spaces.

Contribution

The paper proposes a novel perturbation-based regularization technique, SDARTS, that stabilizes DARTS by smoothing the loss landscape and implicitly regularizing the Hessian norm.

Findings

SDARTS improves stability of DARTS-based methods.

Enhanced performance across various datasets and search spaces.

Mathematical analysis shows regularization of the Hessian norm.

Abstract

Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and largely reduces the search cost. However, its stability has been challenged for yielding deteriorating architectures as the search proceeds. We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. Based on this observation, we propose a perturbation-based regularization - SmoothDARTS (SDARTS), to smooth the loss landscape and improve the generalizability of DARTS-based methods. In particular, our new formulations stabilize DARTS-based methods by either random smoothing or adversarial attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the effectiveness of our approach and due to the improved stability, we achieve performance gain across various search spaces on 4 datasets. Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.