Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator

TL;DR

This paper introduces SGNAS, a novel one-shot NAS framework that uses an architecture generator and a unified supernet to efficiently generate multiple architectures for different hardware constraints with minimal search time.

Contribution

The paper proposes a new architecture generator and a unified supernet to significantly improve the efficiency and flexibility of one-shot NAS, enabling rapid generation of architectures for multiple constraints.

Findings

SGNAS reduces search time to 5 GPU hours for multiple constraints.

Generated architectures achieve 77.1% top-1 accuracy on ImageNet.

SGNAS is 4 times faster than previous state-of-the-art methods.

Abstract

In one-shot NAS, sub-networks need to be searched from the supernet to meet different hardware constraints. However, the search cost is high and $N$ times of searches are needed for $N$ different constraints. In this work, we propose a novel search strategy called architecture generator to search sub-networks by generating them, so that the search process can be much more efficient and flexible. With the trained architecture generator, given target hardware constraints as the input, $N$ good architectures can be generated for $N$ constraints by just one forward pass without re-searching and supernet retraining. Moreover, we propose a novel single-path supernet, called unified supernet, to further improve search efficiency and reduce GPU memory consumption of the architecture generator. With the architecture generator and the unified supernet, we propose a flexible and efficient one-shot NAS framework, called Searching by Generating NAS (SGNAS). With the pre-trained supernt, the search time of SGNAS for $N$ different hardware constraints is only 5 GPU hours, which is $4N$ times faster than previous SOTA single-path methods. After training from scratch, the top1-accuracy of SGNAS on ImageNet is 77.1%, which is comparable with the SOTAs. The code is available at: https://github.com/eric8607242/SGNAS.