Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

TL;DR

Pre-training neural architecture representations in an unsupervised manner enhances the efficiency and effectiveness of neural architecture search by better clustering similar architectures and providing smoother latent space transitions.

Contribution

This work demonstrates that unsupervised pre-training of architecture representations improves NAS performance and offers better clustering of similar architectures, reducing search bias.

Findings

Unsupervised pre-training improves NAS efficiency.

Better clustering of similar architectures in latent space.

Smoother transitions in architecture search space.

Abstract

Existing Neural Architecture Search (NAS) methods either encode neural architectures using discrete encodings that do not scale well, or adopt supervised learning-based methods to jointly learn architecture representations and optimize architecture search on such representations which incurs search bias. Despite the widespread use, architecture representations learned in NAS are still poorly understood. We observe that the structural properties of neural architectures are hard to preserve in the latent space if architecture representation learning and search are coupled, resulting in less effective search performance. In this work, we find empirically that pre-training architecture representations using only neural architectures without their accuracies as labels considerably improve the downstream architecture search efficiency. To explain these observations, we visualize how unsupervised architecture representation learning better encourages neural architectures with similar connections and operators to cluster together. This helps to map neural architectures with similar performance to the same regions in the latent space and makes the transition of architectures in the latent space relatively smooth, which considerably benefits diverse downstream search strategies.