PredNAS: A Universal and Sample Efficient Neural Architecture Search Framework

TL;DR

PredNAS introduces a universal, sample-efficient neural architecture search framework that uses a neural predictor and gradient ascent to optimize architectures, achieving state-of-the-art results with minimal training data.

Contribution

The paper proposes PredNAS, a NAS framework that effectively utilizes a neural predictor and gradient-based optimization, requiring fewer samples than existing methods.

Findings

Achieves state-of-the-art NAS performance with fewer than 100 samples.

Successfully applied to large-scale tasks like ImageNet and MSCOCO.

Explores novel architectures with competitive performance under computational constraints.

Abstract

In this paper, we present a general and effective framework for Neural Architecture Search (NAS), named PredNAS. The motivation is that given a differentiable performance estimation function, we can directly optimize the architecture towards higher performance by simple gradient ascent. Specifically, we adopt a neural predictor as the performance predictor. Surprisingly, PredNAS can achieve state-of-the-art performances on NAS benchmarks with only a few training samples (less than 100). To validate the universality of our method, we also apply our method on large-scale tasks and compare our method with RegNet on ImageNet and YOLOX on MSCOCO. The results demonstrate that our PredNAS can explore novel architectures with competitive performances under specific computational complexity constraints.