Graph HyperNetworks for Neural Architecture Search

TL;DR

This paper introduces Graph HyperNetworks (GHNs) that efficiently generate neural network weights based on architecture topology, enabling faster neural architecture search and improved speed-accuracy tradeoffs.

Contribution

The paper presents GHNs that model architecture topology for rapid weight generation, significantly reducing NAS search time and improving performance over manual designs.

Findings

GHNs enable nearly 10x faster NAS on CIFAR-10 and ImageNet.

GHNs produce more accurate performance predictions than traditional hypernetworks.

GHNs discover networks with superior speed-accuracy tradeoffs in anytime prediction settings.

Abstract

Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Graph HyperNetwork (GHN) to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network. GHNs model the topology of an architecture and therefore can predict network performance more accurately than regular hypernetworks and premature early stopping. To perform NAS, we randomly sample architectures and use the validation accuracy of networks with GHN generated weights as the surrogate search signal. GHNs are fast -- they can search nearly 10 times faster than other random search methods on CIFAR-10 and ImageNet. GHNs can be further extended to the anytime prediction setting, where they have found networks with better speed-accuracy tradeoff than the state-of-the-art manual designs.