Towards Self-supervised and Weight-preserving Neural Architecture Search

TL;DR

This paper introduces SSWP-NAS, a self-supervised, weight-preserving neural architecture search method that simplifies the NAS process, achieves state-of-the-art results without manual labels, and improves semi-supervised learning performance.

Contribution

It proposes a one-stage, proxy-free NAS framework that uses self-supervision and retains weights, streamlining deployment and enhancing semi-supervised learning.

Findings

Achieves state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet.

Outperforms random initialization and pre-training in semi-supervised scenarios.

Simplifies NAS workflow to a one-stage, proxy-free process.

Abstract

Neural architecture search (NAS) algorithms save tremendous labor from human experts. Recent advancements further reduce the computational overhead to an affordable level. However, it is still cumbersome to deploy the NAS techniques in real-world applications due to the fussy procedures and the supervised learning paradigm. In this work, we propose the self-supervised and weight-preserving neural architecture search (SSWP-NAS) as an extension of the current NAS framework by allowing the self-supervision and retaining the concomitant weights discovered during the search stage. As such, we simplify the workflow of NAS to a one-stage and proxy-free procedure. Experiments show that the architectures searched by the proposed framework achieve state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet datasets without using manual labels. Moreover, we show that employing the concomitant weights as initialization consistently outperforms the random initialization and the two-stage weight pre-training method by a clear margin under semi-supervised learning scenarios. Codes are publicly available at https://github.com/LzVv123456/SSWP-NAS.