Improve Ranking Correlation of Super-net through Training Scheme from One-shot NAS to Few-shot NAS

TL;DR

This paper introduces a step-by-step training scheme for super-nets that improves ranking correlation in neural architecture search by transitioning from one-shot to few-shot training, enhancing subnet weight consistency.

Contribution

The paper proposes a novel training scheme that gradually disentangles super-net weights, significantly improving subnet ranking accuracy in NAS.

Findings

Achieved 4th place in CVPR2022 Lightweight NAS Challenge.

Improved ranking correlation of subnets in super-net training.

Demonstrated effectiveness of step-by-step training from one-shot to few-shot NAS.

Abstract

The algorithms of one-shot neural architecture search(NAS) have been widely used to reduce computation consumption. However, because of the interference among the subnets in which weights are shared, the subnets inherited from these super-net trained by those algorithms have poor consistency in precision ranking. To address this problem, we propose a step-by-step training super-net scheme from one-shot NAS to few-shot NAS. In the training scheme, we firstly train super-net in a one-shot way, and then we disentangle the weights of super-net by splitting them into multi-subnets and training them gradually. Finally, our method ranks 4th place in the CVPR2022 3rd Lightweight NAS Challenge Track1. Our code is available at https://github.com/liujiawei2333/CVPR2022-NAS-competition-Track-1-4th-solution.