iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization

TL;DR

The paper introduces iDARTS, an improved neural architecture search method that stabilizes DARTS by node normalization and decorrelated discretization, leading to faster, more robust, and generalizable architecture search results.

Contribution

iDARTS proposes novel node normalization and decorrelated discretization techniques to address instability issues in DARTS, enhancing search stability and performance.

Findings

Achieves 2.25% error on CIFAR-10 within 0.2 GPU-days

Attains 24.7% error on ImageNet within 1.9 GPU-days

Demonstrates improved robustness and generalization over existing DARTS methods

Abstract

Differentiable ARchiTecture Search (DARTS) uses a continuous relaxation of network representation and dramatically accelerates Neural Architecture Search (NAS) by almost thousands of times in GPU-day. However, the searching process of DARTS is unstable, which suffers severe degradation when training epochs become large, thus limiting its application. In this paper, we claim that this degradation issue is caused by the imbalanced norms between different nodes and the highly correlated outputs from various operations. We then propose an improved version of DARTS, namely iDARTS, to deal with the two problems. In the training phase, it introduces node normalization to maintain the norm balance. In the discretization phase, the continuous architecture is approximated based on the similarity between the outputs of the node and the decorrelated operations rather than the values of the architecture parameters. Extensive evaluation is conducted on CIFAR-10 and ImageNet, and the error rates of 2.25\% and 24.7\% are reported within 0.2 and 1.9 GPU-day for architecture search respectively, which shows its effectiveness. Additional analysis also reveals that iDARTS has the advantage in robustness and generalization over other DARTS-based counterparts.