Dynamical Isometry based Rigorous Fair Neural Architecture Search

TL;DR

This paper introduces a neural architecture search method based on dynamical isometry, which enhances fairness and interpretability in module evaluation, leading to state-of-the-art accuracy and stable training.

Contribution

The paper proposes a novel NAS algorithm leveraging dynamical isometry and mean field theory for fair and rigorous module evaluation, improving search reliability.

Findings

Achieves state-of-the-art top-1 accuracy on ImageNet.

Ensures fair module evaluation through Jacobian conditioning.

Provides stable training without sacrificing performance.

Abstract

Recently, the weight-sharing technique has significantly speeded up the training and evaluation procedure of neural architecture search. However, most existing weight-sharing strategies are solely based on experience or observation, which makes the searching results lack interpretability and rationality. In addition, due to the negligence of fairness, current methods are prone to make misjudgments in module evaluation. To address these problems, we propose a novel neural architecture search algorithm based on dynamical isometry. We use the fix point analysis method in the mean field theory to analyze the dynamics behavior in the steady state random neural network, and how dynamic isometry guarantees the fairness of weight-sharing based NAS. Meanwhile, we prove that our module selection strategy is rigorous fair by estimating the generalization error of all modules with well-conditioned Jacobian. Extensive experiments show that, with the same size, the architecture searched by the proposed method can achieve state-of-the-art top-1 validation accuracy on ImageNet classification. In addition, we demonstrate that our method is able to achieve better and more stable training performance without loss of generality.