AutoSpace: Neural Architecture Search with Less Human Interference

TL;DR

AutoSpace introduces a differentiable evolutionary framework to automate neural architecture search space design, reducing human effort and improving model performance on ImageNet under mobile constraints.

Contribution

It proposes a novel differentiable evolutionary method to automatically optimize NAS search spaces, enhancing efficiency and adaptability across computational budgets.

Findings

Achieves 77.8% top-1 accuracy on ImageNet with mobile constraints.

Outperforms EfficientNet-B0 by 0.7% in accuracy.

Significantly improves NAS performance using learned search spaces.

Abstract

Current neural architecture search (NAS) algorithms still require expert knowledge and effort to design a search space for network construction. In this paper, we consider automating the search space design to minimize human interference, which however faces two challenges: the explosive complexity of the exploration space and the expensive computation cost to evaluate the quality of different search spaces. To solve them, we propose a novel differentiable evolutionary framework named AutoSpace, which evolves the search space to an optimal one with following novel techniques: a differentiable fitness scoring function to efficiently evaluate the performance of cells and a reference architecture to speedup the evolution procedure and avoid falling into sub-optimal solutions. The framework is generic and compatible with additional computational constraints, making it feasible to learn specialized search spaces that fit different computational budgets. With the learned search space, the performance of recent NAS algorithms can be improved significantly compared with using previously manually designed spaces. Remarkably, the models generated from the new search space achieve 77.8% top-1 accuracy on ImageNet under the mobile setting (MAdds < 500M), out-performing previous SOTA EfficientNet-B0 by 0.7%. All codes will be made public.