WPNAS: Neural Architecture Search by jointly using Weight Sharing and Predictor

TL;DR

This paper introduces WPNAS, a neural architecture search method that combines weight sharing and a predictor to evaluate architectures more accurately, leading to state-of-the-art results across multiple datasets and search spaces.

Contribution

It proposes a unified framework that jointly uses weight sharing and a predictor, including a novel weakly weight sharing technique with HyperNet to improve evaluation accuracy.

Findings

Achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet.

Effectively combines direct evaluation, prediction, and cost for architecture assessment.

Introduces a weakly weight sharing method to mitigate weight sharing side effects.

Abstract

Weight sharing based and predictor based methods are two major types of fast neural architecture search methods. In this paper, we propose to jointly use weight sharing and predictor in a unified framework. First, we construct a SuperNet in a weight-sharing way and probabilisticly sample architectures from the SuperNet. To increase the correctness of the evaluation of architectures, besides direct evaluation using the inherited weights, we further apply a few-shot predictor to assess the architecture on the other hand. The final evaluation of the architecture is the combination of direct evaluation, the prediction from the predictor and the cost of the architecture. We regard the evaluation as a reward and apply a self-critical policy gradient approach to update the architecture probabilities. To further reduce the side effects of weight sharing, we propose a weakly weight sharing method by introducing another HyperNet. We conduct experiments on datasets including CIFAR-10, CIFAR-100 and ImageNet under NATS-Bench, DARTS and MobileNet search space. The proposed WPNAS method achieves state-of-the-art performance on these datasets.