Extensible Proxy for Efficient NAS

TL;DR

This paper introduces Eproxy, an extensible, near-zero-cost neural architecture performance predictor using self-supervised, few-shot training, adaptable to various search spaces and downstream tasks, significantly reducing NAS computational costs.

Contribution

The paper proposes Eproxy, a novel extensible proxy for NAS that employs a barrier layer and discrete proxy search to adapt to different tasks and search spaces with minimal training.

Findings

Eproxy achieves near-zero-cost performance prediction.

Eproxy is adaptable to multiple search spaces and downstream tasks.

Experimental results confirm the effectiveness of Eproxy and DPS.

Abstract

Neural Architecture Search (NAS) has become a de facto approach in the recent trend of AutoML to design deep neural networks (DNNs). Efficient or near-zero-cost NAS proxies are further proposed to address the demanding computational issues of NAS, where each candidate architecture network only requires one iteration of backpropagation. The values obtained from the proxies are considered the predictions of architecture performance on downstream tasks. However, two significant drawbacks hinder the extended usage of Efficient NAS proxies. (1) Efficient proxies are not adaptive to various search spaces. (2) Efficient proxies are not extensible to multi-modality downstream tasks. Based on the observations, we design a Extensible proxy (Eproxy) that utilizes self-supervised, few-shot training (i.e., 10 iterations of backpropagation) which yields near-zero costs. The key component that makes Eproxy efficient is an untrainable convolution layer termed barrier layer that add the non-linearities to the optimization spaces so that the Eproxy can discriminate the performance of architectures in the early stage. Furthermore, to make Eproxy adaptive to different downstream tasks/search spaces, we propose a Discrete Proxy Search (DPS) to find the optimized training settings for Eproxy with only handful of benchmarked architectures on the target tasks. Our extensive experiments confirm the effectiveness of both Eproxy and Eproxy+DPS. Code is available at https://github.com/leeyeehoo/GenNAS-Zero.