Generalized Global Ranking-Aware Neural Architecture Ranker for Efficient Image Classifier Search

TL;DR

This paper introduces a global ranking-aware neural architecture predictor called NAR that improves NAS efficiency by classifying architectures into quality tiers, enabling better generalization and simplified search.

Contribution

The paper proposes NAR, a global quality distribution-based predictor for NAS that enhances generalization and simplifies the search process by directly sampling from quality tiers.

Findings

NAR outperforms state-of-the-art methods on NAS-Bench-101.

NAR effectively generalizes across different datasets like CIFAR-10, CIFAR-100, and ImageNet-16-120.

Sampling based on quality tiers finds top-performing architectures efficiently.

Abstract

Neural Architecture Search (NAS) is a powerful tool for automating effective image processing DNN designing. The ranking has been advocated to design an efficient performance predictor for NAS. The previous contrastive method solves the ranking problem by comparing pairs of architectures and predicting their relative performance. However, it only focuses on the rankings between two involved architectures and neglects the overall quality distributions of the search space, which may suffer generalization issues. A predictor, namely Neural Architecture Ranker (NAR) which concentrates on the global quality tier of specific architecture, is proposed to tackle such problems caused by the local perspective. The NAR explores the quality tiers of the search space globally and classifies each individual to the tier they belong to according to its global ranking. Thus, the predictor gains the knowledge of the performance distributions of the search space which helps to generalize its ranking ability to the datasets more easily. Meanwhile, the global quality distribution facilitates the search phase by directly sampling candidates according to the statistics of quality tiers, which is free of training a search algorithm, e.g., Reinforcement Learning (RL) or Evolutionary Algorithm (EA), thus it simplifies the NAS pipeline and saves the computational overheads. The proposed NAR achieves better performance than the state-of-the-art methods on two widely used datasets for NAS research. On the vast search space of NAS-Bench-101, the NAR easily finds the architecture with top 0.01$\unicode{x2030}$ performance only by sampling. It also generalizes well to different image datasets of NAS-Bench-201, i.e., CIFAR-10, CIFAR-100, and ImageNet-16-120 by identifying the optimal architectures for each of them.