Revisiting Training-free NAS Metrics: An Efficient Training-based Method

TL;DR

This paper critically examines training-free NAS metrics, highlighting the importance of the number of parameters (Param), and introduces a lightweight training-based metric that outperforms existing methods with lower search costs.

Contribution

It reveals the reliance of recent training-free metrics on Param and proposes a new training-based metric with weak Param correlation for more effective neural architecture search.

Findings

Param is a surprisingly effective metric overlooked in previous work.

Recent training-free metrics depend heavily on Param information.

The proposed method achieves competitive results on ImageNet with only 2.6 GPU hours.

Abstract

Recent neural architecture search (NAS) works proposed training-free metrics to rank networks which largely reduced the search cost in NAS. In this paper, we revisit these training-free metrics and find that: (1) the number of parameters (\#Param), which is the most straightforward training-free metric, is overlooked in previous works but is surprisingly effective, (2) recent training-free metrics largely rely on the \#Param information to rank networks. Our experiments show that the performance of recent training-free metrics drops dramatically when the \#Param information is not available. Motivated by these observations, we argue that metrics less correlated with the \#Param are desired to provide additional information for NAS. We propose a light-weight training-based metric which has a weak correlation with the \#Param while achieving better performance than training-free metrics at a lower search cost. Specifically, on DARTS search space, our method completes searching directly on ImageNet in only 2.6 GPU hours and achieves a top-1/top-5 error rate of 24.1\%/7.1\%, which is competitive among state-of-the-art NAS methods. Codes are available at \url{https://github.com/taoyang1122/Revisit_TrainingFree_NAS}