Multi-trial Neural Architecture Search with Lottery Tickets

TL;DR

This paper introduces a new neural architecture search method called MENAS that efficiently explores a broader search space by leveraging lottery tickets and early pruning, leading to state-of-the-art results on multiple datasets.

Contribution

It proposes a novel search space MobileNet3-MT and an efficient evolutionary NAS method MENAS that accelerates search via pruning and lottery tickets.

Findings

MENAS achieves state-of-the-art performance on ImageNet-1K.

The new search space allows for more potential model candidates.

Efficient search process reduces computational cost.

Abstract

Neural architecture search (NAS) has brought significant progress in recent image recognition tasks. Most existing NAS methods apply restricted search spaces, which limits the upper-bound performance of searched models. To address this issue, we propose a new search space named MobileNet3-MT. By reducing human-prior knowledge in omni dimensions of networks, MobileNet3-MT accommodates more potential candidates. For searching in this challenging search space, we present an efficient Multi-trial Evolution-based NAS method termed MENAS. Specifically, we accelerate the evolutionary search process by gradually pruning models in the population. Each model is trained with an early stop and replaced by its Lottery Tickets (the explored optimal pruned network).In this way, the full training pipeline of cumbersome networks is prevented and more efficient networks are automatically generated. Extensive experimental results on ImageNet-1K, CIFAR-10, and CIFAR-100 demonstrate that MENAS achieves state-of-the-art performance.