Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Search

TL;DR

This paper introduces PBT-NAS, a neural architecture search method that improves architectures during training by mixing well-performing networks with a shrink-perturb technique, leading to superior results without retraining.

Contribution

The paper proposes PBT-NAS, an adaptation of Population Based Training that enhances neural architecture search through architecture mixing and weight inheritance, enabling direct use of trained networks.

Findings

PBT-NAS outperforms random search and mutation-based PBT on challenging tasks.

Networks created by PBT-NAS can be used directly without retraining.

PBT-NAS is highly parallelizable and effective for NAS.

Abstract

In this work, we show that simultaneously training and mixing neural networks is a promising way to conduct Neural Architecture Search (NAS). For hyperparameter optimization, reusing the partially trained weights allows for efficient search, as was previously demonstrated by the Population Based Training (PBT) algorithm. We propose PBT-NAS, an adaptation of PBT to NAS where architectures are improved during training by replacing poorly-performing networks in a population with the result of mixing well-performing ones and inheriting the weights using the shrink-perturb technique. After PBT-NAS terminates, the created networks can be directly used without retraining. PBT-NAS is highly parallelizable and effective: on challenging tasks (image generation and reinforcement learning) PBT-NAS achieves superior performance compared to baselines (random search and mutation-based PBT).