Homogeneous Architecture Augmentation for Neural Predictor

TL;DR

This paper introduces HAAP, a novel homogeneous architecture augmentation method that enhances neural predictors for NAS by generating more training data efficiently, leading to improved performance with less data.

Contribution

The paper proposes a homogeneous architecture augmentation algorithm and a one-hot encoding strategy to improve neural predictor training in NAS.

Findings

HAAP outperforms state-of-the-art methods on NAS-Benchmark-101 and NAS-Bench-201.

HAAP requires significantly less training data to achieve high prediction accuracy.

The homogeneous augmentation method is shown to be universal across different datasets.

Abstract

Neural Architecture Search (NAS) can automatically design well-performed architectures of Deep Neural Networks (DNNs) for the tasks at hand. However, one bottleneck of NAS is the prohibitively computational cost largely due to the expensive performance evaluation. The neural predictors can directly estimate the performance without any training of the DNNs to be evaluated, thus have drawn increasing attention from researchers. Despite their popularity, they also suffer a severe limitation: the shortage of annotated DNN architectures for effectively training the neural predictors. In this paper, we proposed Homogeneous Architecture Augmentation for Neural Predictor (HAAP) of DNN architectures to address the issue aforementioned. Specifically, a homogeneous architecture augmentation algorithm is proposed in HAAP to generate sufficient training data taking the use of homogeneous representation. Furthermore, the one-hot encoding strategy is introduced into HAAP to make the representation of DNN architectures more effective. The experiments have been conducted on both NAS-Benchmark-101 and NAS-Bench-201 dataset. The experimental results demonstrate that the proposed HAAP algorithm outperforms the state of the arts compared, yet with much less training data. In addition, the ablation studies on both benchmark datasets have also shown the universality of the homogeneous architecture augmentation.