GraphPNAS: Learning Distribution of Good Neural Architectures via Deep Graph Generative Models

TL;DR

GraphPNAS introduces a deep graph generative model for neural architecture search, learning a distribution of high-performing architectures to improve flexibility and efficiency over traditional point-estimation methods.

Contribution

The paper proposes GraphPNAS, a novel GNN-based generative model for NAS that captures architecture topologies and relations, enabling probabilistic search and outperforming existing methods.

Findings

GraphPNAS outperforms RNN-based generators in NAS tasks.

It achieves comparable or better results than state-of-the-art NAS methods.

Effective on diverse search spaces including complex RandWire.

Abstract

Neural architectures can be naturally viewed as computational graphs. Motivated by this perspective, we, in this paper, study neural architecture search (NAS) through the lens of learning random graph models. In contrast to existing NAS methods which largely focus on searching for a single best architecture, i.e, point estimation, we propose GraphPNAS a deep graph generative model that learns a distribution of well-performing architectures. Relying on graph neural networks (GNNs), our GraphPNAS can better capture topologies of good neural architectures and relations between operators therein. Moreover, our graph generator leads to a learnable probabilistic search method that is more flexible and efficient than the commonly used RNN generator and random search methods. Finally, we learn our generator via an efficient reinforcement learning formulation for NAS. To assess the effectiveness of our GraphPNAS, we conduct extensive experiments on three search spaces, including the challenging RandWire on TinyImageNet, ENAS on CIFAR10, and NAS-Bench-101/201. The complexity of RandWire is significantly larger than other search spaces in the literature. We show that our proposed graph generator consistently outperforms RNN-based one and achieves better or comparable performances than state-of-the-art NAS methods.