ASAP: Architecture Search, Anneal and Prune

TL;DR

This paper introduces a differentiable neural architecture search method that uses annealing and gradual pruning to efficiently find high-performing models with minimal GPU time, achieving competitive accuracy.

Contribution

It proposes a novel NAS approach combining annealing and gradual pruning within a differentiable search space for efficient architecture optimization.

Findings

Achieves 1.68% error on CIFAR-10 with 0.2 GPU days.

Reduces search cost compared to previous NAS methods.

Demonstrates effective architecture search on multiple vision datasets.

Abstract

Automatic methods for Neural Architecture Search (NAS) have been shown to produce state-of-the-art network models. Yet, their main drawback is the computational complexity of the search process. As some primal methods optimized over a discrete search space, thousands of days of GPU were required for convergence. A recent approach is based on constructing a differentiable search space that enables gradient-based optimization, which reduces the search time to a few days. While successful, it still includes some noncontinuous steps, e.g., the pruning of many weak connections at once. In this paper, we propose a differentiable search space that allows the annealing of architecture weights, while gradually pruning inferior operations. In this way, the search converges to a single output network in a continuous manner. Experiments on several vision datasets demonstrate the effectiveness of our method with respect to the search cost and accuracy of the achieved model. Specifically, with $0.2$ GPU search days we achieve an error rate of $1.68\%$ on CIFAR-10.