Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

TL;DR

This paper reviews neural architecture search (NAS), focusing on weight-sharing methods, highlighting the optimization gap challenge, and categorizing approaches to bridge this gap, with insights on future directions in NAS and AutoML.

Contribution

It provides a comprehensive review of weight-sharing NAS methods, analyzes their strategies to address the optimization gap, and discusses future research directions in NAS and AutoML.

Findings

Weight-sharing NAS methods are faster but face stability issues.

The optimization gap between super-network and sub-architectures is a major challenge.

Categorization of approaches to bridge the optimization gap.

Abstract

Neural architecture search (NAS) has attracted increasing attentions in both academia and industry. In the early age, researchers mostly applied individual search methods which sample and evaluate the candidate architectures separately and thus incur heavy computational overheads. To alleviate the burden, weight-sharing methods were proposed in which exponentially many architectures share weights in the same super-network, and the costly training procedure is performed only once. These methods, though being much faster, often suffer the issue of instability. This paper provides a literature review on NAS, in particular the weight-sharing methods, and points out that the major challenge comes from the optimization gap between the super-network and the sub-architectures. From this perspective, we summarize existing approaches into several categories according to their efforts in bridging the gap, and analyze both advantages and disadvantages of these methodologies. Finally, we share our opinions on the future directions of NAS and AutoML. Due to the expertise of the authors, this paper mainly focuses on the application of NAS to computer vision problems and may bias towards the work in our group.