Generative Adversarial Neural Architecture Search

TL;DR

This paper introduces GA-NAS, a generative adversarial approach to neural architecture search that offers theoretical guarantees, improved stability, and the ability to explore large search spaces efficiently, outperforming existing methods.

Contribution

GA-NAS is a novel NAS method combining importance sampling and adversarial learning with reinforcement, providing convergence guarantees and enhanced reproducibility.

Findings

GA-NAS outperforms existing NAS benchmarks.

It effectively handles search constraints and spaces.

Can improve baseline models like EfficientNet.

Abstract

Despite the empirical success of neural architecture search (NAS) in deep learning applications, the optimality, reproducibility and cost of NAS schemes remain hard to assess. In this paper, we propose Generative Adversarial NAS (GA-NAS) with theoretically provable convergence guarantees, promoting stability and reproducibility in neural architecture search. Inspired by importance sampling, GA-NAS iteratively fits a generator to previously discovered top architectures, thus increasingly focusing on important parts of a large search space. Furthermore, we propose an efficient adversarial learning approach, where the generator is trained by reinforcement learning based on rewards provided by a discriminator, thus being able to explore the search space without evaluating a large number of architectures. Extensive experiments show that GA-NAS beats the best published results under several cases on three public NAS benchmarks. In the meantime, GA-NAS can handle ad-hoc search constraints and search spaces. We show that GA-NAS can be used to improve already optimized baselines found by other NAS methods, including EfficientNet and ProxylessNAS, in terms of ImageNet accuracy or the number of parameters, in their original search space.