Evolving Search Space for Neural Architecture Search

TL;DR

This paper introduces a Neural Search-space Evolution (NSE) method that iteratively refines the search space for neural architecture search, leading to improved performance and reduced need for manual search space design.

Contribution

The paper proposes a novel NSE scheme that automatically evolves the search space and incorporates a learnable multi-branch architecture, achieving state-of-the-art results without manual search space engineering.

Findings

Achieved 77.3% top-1 accuracy on ImageNet with 333M FLOPs.

Outperformed previous auto-generated architectures without knowledge distillation.

Surpassed prior mobile models under latency constraints.

Abstract

The automation of neural architecture design has been a coveted alternative to human experts. Recent works have small search space, which is easier to optimize but has a limited upper bound of the optimal solution. Extra human design is needed for those methods to propose a more suitable space with respect to the specific task and algorithm capacity. To further enhance the degree of automation for neural architecture search, we present a Neural Search-space Evolution (NSE) scheme that iteratively amplifies the results from the previous effort by maintaining an optimized search space subset. This design minimizes the necessity of a well-designed search space. We further extend the flexibility of obtainable architectures by introducing a learnable multi-branch setting. By employing the proposed method, a consistent performance gain is achieved during a progressive search over upcoming search spaces. We achieve 77.3% top-1 retrain accuracy on ImageNet with 333M FLOPs, which yielded a state-of-the-art performance among previous auto-generated architectures that do not involve knowledge distillation or weight pruning. When the latency constraint is adopted, our result also performs better than the previous best-performing mobile models with a 77.9% Top-1 retrain accuracy.