Evaluating Efficient Performance Estimators of Neural Architectures

TL;DR

This paper thoroughly evaluates one-shot and zero-shot estimators for neural architecture performance, revealing biases and variances, and offers insights to improve their reliability in neural architecture search.

Contribution

It provides an extensive assessment of OSEs and ZSEs across multiple benchmarks, analyzing their biases, variances, and proposing mitigation strategies for better estimations.

Findings

OSEs and ZSEs exhibit biases and variances affecting estimation quality.

Analysis reveals the correlation gap in OSEs and suggests mitigation approaches.

Framework aids future research in designing more reliable architecture estimators.

Abstract

Conducting efficient performance estimations of neural architectures is a major challenge in neural architecture search (NAS). To reduce the architecture training costs in NAS, one-shot estimators (OSEs) amortize the architecture training costs by sharing the parameters of one "supernet" between all architectures. Recently, zero-shot estimators (ZSEs) that involve no training are proposed to further reduce the architecture evaluation cost. Despite the high efficiency of these estimators, the quality of such estimations has not been thoroughly studied. In this paper, we conduct an extensive and organized assessment of OSEs and ZSEs on five NAS benchmarks: NAS-Bench-101/201/301, and NDS ResNet/ResNeXt-A. Specifically, we employ a set of NAS-oriented criteria to study the behavior of OSEs and ZSEs and reveal that they have certain biases and variances. After analyzing how and why the OSE estimations are unsatisfying, we explore how to mitigate the correlation gap of OSEs from several perspectives. Through our analysis, we give out suggestions for future application and development of efficient architecture performance estimators. Furthermore, the analysis framework proposed in our work could be utilized in future research to give a more comprehensive understanding of newly designed architecture performance estimators. All codes are available at https://github.com/walkerning/aw_nas.