Enabling NAS with Automated Super-Network Generation

TL;DR

BootstrapNAS automates super-network generation from pre-trained models, enabling efficient NAS and resulting in subnetworks that outperform original models, facilitating resource-constrained deployment.

Contribution

We introduce BootstrapNAS, a framework that automatically creates super-networks from arbitrary models, simplifying NAS and improving subnetwork performance.

Findings

Super-networks generated outperform original pre-trained models.

Framework works with arbitrary architectures including custom designs.

Results are reproducible with publicly available super-networks.

Abstract

Recent Neural Architecture Search (NAS) solutions have produced impressive results training super-networks and then deriving subnetworks, a.k.a. child models that outperform expert-crafted models from a pre-defined search space. Efficient and robust subnetworks can be selected for resource-constrained edge devices, allowing them to perform well in the wild. However, constructing super-networks for arbitrary architectures is still a challenge that often prevents the adoption of these approaches. To address this challenge, we present BootstrapNAS, a software framework for automatic generation of super-networks for NAS. BootstrapNAS takes a pre-trained model from a popular architecture, e.g., ResNet- 50, or from a valid custom design, and automatically creates a super-network out of it, then uses state-of-the-art NAS techniques to train the super-network, resulting in subnetworks that significantly outperform the given pre-trained model. We demonstrate the solution by generating super-networks from arbitrary model repositories and make available the resulting super-networks for reproducibility of the results.