Efficacy of Neural Prediction-Based Zero-Shot NAS

TL;DR

This paper introduces a novel zero-shot neural architecture search method using Fourier sine encodings and an MLP to evaluate architectures across different search spaces, outperforming previous graph-based methods.

Contribution

It proposes a new deep learning-based zero-shot NAS approach with learnable Fourier encodings that generalize across multiple search spaces, improving performance prediction accuracy.

Findings

Outperforms previous graph convolutional methods on NAS-Bench-201.

Achieves higher correlation and convergence rate in architecture ranking.

Features are transferable across different NAS benchmarks.

Abstract

In prediction-based Neural Architecture Search (NAS), performance indicators derived from graph convolutional networks have shown remarkable success. These indicators, achieved by representing feed-forward structures as component graphs through one-hot encoding, face a limitation: their inability to evaluate architecture performance across varying search spaces. In contrast, handcrafted performance indicators (zero-shot NAS), which use the same architecture with random initialization, can generalize across multiple search spaces. Addressing this limitation, we propose a novel approach for zero-shot NAS using deep learning. Our method employs Fourier sum of sines encoding for convolutional kernels, enabling the construction of a computational feed-forward graph with a structure similar to the architecture under evaluation. These encodings are learnable and offer a comprehensive view of the architecture's topological information. An accompanying multi-layer perceptron (MLP) then ranks these architectures based on their encodings. Experimental results show that our approach surpasses previous methods using graph convolutional networks in terms of correlation on the NAS-Bench-201 dataset and exhibits a higher convergence rate. Moreover, our extracted feature representation trained on each NAS benchmark is transferable to other NAS benchmarks, showing promising generalizability across multiple search spaces. The code is available at: https://github.com/minh1409/DFT-NPZS-NAS