Understanding and Robustifying Differentiable Architecture Search

TL;DR

This paper investigates the failure modes of Differentiable Architecture Search (DARTS), identifies causes related to high curvature solutions, and proposes regularization techniques to improve its robustness and generalization across various tasks and domains.

Contribution

The paper reveals why DARTS can produce poor architectures, introduces regularization methods to mitigate this issue, and demonstrates improved robustness across multiple search spaces and tasks.

Findings

Regularization reduces high curvature solutions in DARTS.

Proposed variations of DARTS improve robustness and generalization.

Results are consistent across image classification, disparity estimation, and language modeling.

Abstract

Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.