Adaptive Channel Allocation for Robust Differentiable Architecture Search

TL;DR

This paper introduces an Adaptive Channel Allocation strategy for differentiable architecture search that enhances robustness and stability by implicitly managing skip connections through channel importance ranking, without human priors or hyper-parameters.

Contribution

It proposes a novel ACA strategy that implicitly refills unimportant channels during evaluation, improving DARTS robustness without manual intervention.

Findings

ACA improves robustness across datasets and DARTS variants.

ACA effectively mitigates collapse issues in long training.

The method is general-purpose and hyper-parameter free.

Abstract

Differentiable ARchiTecture Search (DARTS) has attracted much attention due to its simplicity and significant improvement in efficiency. However, the excessive accumulation of the skip connection, when training epochs become large, makes it suffer from weak stability and low robustness, thus limiting its practical applications. Many works have attempted to restrict the accumulation of skip connections by indicators or manual design. These methods, however, are susceptible to human priors and hyper-parameters. In this work, we suggest a more subtle and direct approach that no longer explicitly searches for skip connections in the search stage, based on the paradox that skip connections were proposed to guarantee the performance of very deep networks, but the networks in the search stage of differentiable architecture search are actually very shallow. Instead, by introducing channel importance ranking and channel allocation strategy, the skip connections are implicitly searched and automatically refilled unimportant channels in the evaluation stage. Our method, dubbed Adaptive Channel Allocation (ACA) strategy, is a general-purpose approach for differentiable architecture search, which universally works in DARTS variants without introducing human priors, indicators, or hyper-parameters. Extensive experiments on various datasets and DARTS variants verify that the ACA strategy is the most effective one among existing methods in improving robustness and dealing with the collapse issue when training epochs become large.