Entropic Score metric: Decoupling Topology and Size in Training-free NAS

TL;DR

This paper introduces Entropic Score, a training-free metric for neural architecture search that efficiently estimates model expressivity, enabling rapid design of high-performance models for edge devices without extensive training.

Contribution

The paper proposes a novel Entropic Score metric and a cyclic search algorithm to decouple and optimize model topology and size in training-free NAS.

Findings

Entropic Score effectively estimates network topology.

Combining Entropic Score with LogSynflow improves size search.

Achieves state-of-the-art NAS results on ImageNet in under 1 GPU hour.

Abstract

Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typical of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and computational resources to generate accurate models. To overcome these limits, this paper contributes with: i) a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise entropy of its activations; ii) a cyclic search algorithm to separately yet synergistically search model size and topology. Entropic Score shows remarkable ability in searching for the topology of the network, and a proper combination with LogSynflow, to search for model size, yields superior capability to completely design high-performance Hybrid Transformers for edge applications in less than 1 GPU hour, resulting in the fastest and most accurate NAS method for ImageNet classification.