BN-NAS: Neural Architecture Search with Batch Normalization

TL;DR

BN-NAS introduces a novel approach to neural architecture search that leverages Batch Normalization to predict subnet performance early and trains only BN parameters, greatly reducing search time without sacrificing accuracy.

Contribution

The paper proposes BN-NAS, a method that accelerates NAS by using BN-based indicators for early performance prediction and training only BN parameters during supernet training.

Findings

Supernet training time reduced by over 10 times.

Subnet evaluation time shortened by more than 600,000 times.

Maintains accuracy despite significant speedups.

Abstract

We present BN-NAS, neural architecture search with Batch Normalization (BN-NAS), to accelerate neural architecture search (NAS). BN-NAS can significantly reduce the time required by model training and evaluation in NAS. Specifically, for fast evaluation, we propose a BN-based indicator for predicting subnet performance at a very early training stage. The BN-based indicator further facilitates us to improve the training efficiency by only training the BN parameters during the supernet training. This is based on our observation that training the whole supernet is not necessary while training only BN parameters accelerates network convergence for network architecture search. Extensive experiments show that our method can significantly shorten the time of training supernet by more than 10 times and shorten the time of evaluating subnets by more than 600,000 times without losing accuracy.