FR-NAS: Forward-and-Reverse Graph Predictor for Efficient Neural Architecture Search

TL;DR

FR-NAS introduces a novel GNN-based predictor that uses both forward and reverse graph views to efficiently estimate neural network performance, significantly reducing training data requirements and improving prediction accuracy in NAS.

Contribution

The paper presents a new GNN predictor leveraging dual graph views and a custom loss to enhance NAS performance prediction with limited data.

Findings

Achieved 3%-16% improvement in Kendall-tau correlation over existing GNN predictors.

Effectively utilized small datasets (50-400 samples) for accurate performance prediction.

Validated on NAS-Bench-101, NAS-Bench-201, and DARTS search space.

Abstract

Neural Architecture Search (NAS) has emerged as a key tool in identifying optimal configurations of deep neural networks tailored to specific tasks. However, training and assessing numerous architectures introduces considerable computational overhead. One method to mitigating this is through performance predictors, which offer a means to estimate the potential of an architecture without exhaustive training. Given that neural architectures fundamentally resemble Directed Acyclic Graphs (DAGs), Graph Neural Networks (GNNs) become an apparent choice for such predictive tasks. Nevertheless, the scarcity of training data can impact the precision of GNN-based predictors. To address this, we introduce a novel GNN predictor for NAS. This predictor renders neural architectures into vector representations by combining both the conventional and inverse graph views. Additionally, we incorporate a customized training loss within the GNN predictor to ensure efficient utilization of both types of representations. We subsequently assessed our method through experiments on benchmark datasets including NAS-Bench-101, NAS-Bench-201, and the DARTS search space, with a training dataset ranging from 50 to 400 samples. Benchmarked against leading GNN predictors, the experimental results showcase a significant improvement in prediction accuracy, with a 3%--16% increase in Kendall-tau correlation. Source codes are available at https://github.com/EMI-Group/fr-nas.