Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning

TL;DR

This paper introduces a locality-based self-supervised pretraining method for neural architecture search controllers, reducing computational costs by leveraging architecture similarities to improve search efficiency.

Contribution

It proposes a novel self-supervised learning scheme for NAS controllers that enhances architecture representations and integrates metric learning to improve search performance.

Findings

Pretraining improves NAS efficiency and effectiveness.

Adding metric learning loss enhances architecture embeddings.

The method is generally applicable to controller-based NAS.

Abstract

Neural architecture search (NAS) has fostered various fields of machine learning. Despite its prominent dedications, many have criticized the intrinsic limitations of high computational cost. We aim to ameliorate this by proposing a pretraining scheme that can be generally applied to controller-based NAS. Our method, locality-based self-supervised classification task, leverages the structural similarity of network architectures to obtain good architecture representations. We incorporate our method into neural architecture optimization (NAO) to analyze the pretrained embeddings and its effectiveness and highlight that adding metric learning loss brings a favorable impact on NAS. Our code is available at \url{https://github.com/Multi-Objective-NAS/self-supervised-nas}.