A Survey of Supernet Optimization and its Applications: Spatial and Temporal Optimization for Neural Architecture Search

TL;DR

This survey reviews supernet optimization techniques in Neural Architecture Search, focusing on spatial and temporal methods, their benefits, limitations, and applications across different tasks and models.

Contribution

It categorizes and evaluates supernet optimization methods based on spatial and temporal approaches, providing a comprehensive overview of their applications and challenges.

Findings

Supernet optimization enables efficient NAS by training a single over-parameterized network.

Spatial and temporal optimization methods have distinct benefits and limitations.

Applications include transferability, domain generalization, and Transformer models.

Abstract

This survey focuses on categorizing and evaluating the methods of supernet optimization in the field of Neural Architecture Search (NAS). Supernet optimization involves training a single, over-parameterized network that encompasses the search space of all possible network architectures. The survey analyses supernet optimization methods based on their approaches to spatial and temporal optimization. Spatial optimization relates to optimizing the architecture and parameters of the supernet and its subnets, while temporal optimization deals with improving the efficiency of selecting architectures from the supernet. The benefits, limitations, and potential applications of these methods in various tasks and settings, including transferability, domain generalization, and Transformer models, are also discussed.