ASAP: Architecture Search, Anneal and Prune
Asaf Noy, Niv Nayman, Tal Ridnik, Nadav Zamir, Sivan Doveh, Itamar, Friedman, Raja Giryes, and Lihi Zelnik-Manor

TL;DR
This paper introduces a differentiable neural architecture search method that uses annealing and gradual pruning to efficiently find high-performing models with minimal GPU time, achieving competitive accuracy.
Contribution
It proposes a novel NAS approach combining annealing and gradual pruning within a differentiable search space for efficient architecture optimization.
Findings
Achieves 1.68% error on CIFAR-10 with 0.2 GPU days.
Reduces search cost compared to previous NAS methods.
Demonstrates effective architecture search on multiple vision datasets.
Abstract
Automatic methods for Neural Architecture Search (NAS) have been shown to produce state-of-the-art network models. Yet, their main drawback is the computational complexity of the search process. As some primal methods optimized over a discrete search space, thousands of days of GPU were required for convergence. A recent approach is based on constructing a differentiable search space that enables gradient-based optimization, which reduces the search time to a few days. While successful, it still includes some noncontinuous steps, e.g., the pruning of many weak connections at once. In this paper, we propose a differentiable search space that allows the annealing of architecture weights, while gradually pruning inferior operations. In this way, the search converges to a single output network in a continuous manner. Experiments on several vision datasets demonstrate the effectiveness of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Advanced Image and Video Retrieval Techniques
MethodsPruning · Exponential Decay · Cosine Annealing · Cosine Power Annealing · Sigmoid Activation · Tanh Activation · Softmax · Long Short-Term Memory
