RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving

TL;DR

RANK-NOSH introduces a hierarchical scheduling algorithm that significantly reduces architecture training costs in predictor-based NAS by early termination, achieving comparable or better results with about one-fifth of the computation.

Contribution

The paper proposes NOSH, a novel non-uniform successive halving method, and RANK-NOSH, a predictor-based NAS approach that improves efficiency by early stopping and learning to rank.

Findings

Reduces search budget by approximately 5 times.

Achieves competitive or superior performance compared to state-of-the-art methods.

Effective in various search spaces and datasets.

Abstract

Predictor-based algorithms have achieved remarkable performance in the Neural Architecture Search (NAS) tasks. However, these methods suffer from high computation costs, as training the performance predictor usually requires training and evaluating hundreds of architectures from scratch. Previous works along this line mainly focus on reducing the number of architectures required to fit the predictor. In this work, we tackle this challenge from a different perspective - improve search efficiency by cutting down the computation budget of architecture training. We propose NOn-uniform Successive Halving (NOSH), a hierarchical scheduling algorithm that terminates the training of underperforming architectures early to avoid wasting budget. To effectively leverage the non-uniform supervision signals produced by NOSH, we formulate predictor-based architecture search as learning to rank with pairwise comparisons. The resulting method - RANK-NOSH, reduces the search budget by ~5x while achieving competitive or even better performance than previous state-of-the-art predictor-based methods on various spaces and datasets.