DAS: Neural Architecture Search via Distinguishing Activation Score

TL;DR

This paper introduces DAS, a new metric for neural architecture search that decouples existing scores into fundamental components, combined with a fast training strategy and a new dataset, to improve search efficiency and accuracy.

Contribution

Decouples WOT score into atomic metrics, proposes DAS with better combination rules, and introduces a fast training strategy and a new dataset for NAS evaluation.

Findings

DAS improves NAS performance by 1.04x to 1.56x on benchmarks.

Theoretical proof of decoupling WOT score into atomic metrics.

Experimental validation confirms effectiveness of DAS and the training strategy.

Abstract

Neural Architecture Search (NAS) is an automatic technique that can search for well-performed architectures for a specific task. Although NAS surpasses human-designed architecture in many fields, the high computational cost of architecture evaluation it requires hinders its development. A feasible solution is to directly evaluate some metrics in the initial stage of the architecture without any training. NAS without training (WOT) score is such a metric, which estimates the final trained accuracy of the architecture through the ability to distinguish different inputs in the activation layer. However, WOT score is not an atomic metric, meaning that it does not represent a fundamental indicator of the architecture. The contributions of this paper are in three folds. First, we decouple WOT into two atomic metrics which represent the distinguishing ability of the network and the number of activation units, and explore better combination rules named (Distinguishing Activation Score) DAS. We prove the correctness of decoupling theoretically and confirmed the effectiveness of the rules experimentally. Second, in order to improve the prediction accuracy of DAS to meet practical search requirements, we propose a fast training strategy. When DAS is used in combination with the fast training strategy, it yields more improvements. Third, we propose a dataset called Darts-training-bench (DTB), which fills the gap that no training states of architecture in existing datasets. Our proposed method has 1.04$\times$ - 1.56$\times$ improvements on NAS-Bench-101, Network Design Spaces, and the proposed DTB.