Improving Ranking Correlation of Supernet with Candidates Enhancement and Progressive Training

TL;DR

This paper proposes a candidates enhancement and progressive training method to improve the ranking correlation of supernets in one-shot neural architecture search, reducing interference between sub-networks and achieving top performance in a NAS challenge.

Contribution

It introduces a novel progressive training pipeline and supernet redesign to enhance ranking accuracy in NAS, addressing weight-sharing interference.

Findings

Achieved 1st place in CVPR2021 Lightweight NAS Challenge.

Significantly improved ranking correlation of supernet.

Reduces mutual interference between sub-networks.

Abstract

One-shot neural architecture search (NAS) applies weight-sharing supernet to reduce the unaffordable computation overhead of automated architecture designing. However, the weight-sharing technique worsens the ranking consistency of performance due to the interferences between different candidate networks. To address this issue, we propose a candidates enhancement method and progressive training pipeline to improve the ranking correlation of supernet. Specifically, we carefully redesign the sub-networks in the supernet and map the original supernet to a new one of high capacity. In addition, we gradually add narrow branches of supernet to reduce the degree of weight sharing which effectively alleviates the mutual interference between sub-networks. Finally, our method ranks the 1st place in the Supernet Track of CVPR2021 1st Lightweight NAS Challenge.