RankNAS: Efficient Neural Architecture Search by Pairwise Ranking

TL;DR

RankNAS introduces a pairwise ranking approach to neural architecture search, significantly reducing the number of training examples needed and accelerating the search process without relying on performance predictors.

Contribution

It proposes a novel pairwise ranking method for NAS that improves efficiency and incorporates a pruning strategy to focus on promising architectures.

Findings

Achieves high-performance architectures with fewer training examples.

Speeds up NAS by orders of magnitude compared to existing methods.

Effective on machine translation and language modeling tasks.

Abstract

This paper addresses the efficiency challenge of Neural Architecture Search (NAS) by formulating the task as a ranking problem. Previous methods require numerous training examples to estimate the accurate performance of architectures, although the actual goal is to find the distinction between "good" and "bad" candidates. Here we do not resort to performance predictors. Instead, we propose a performance ranking method (RankNAS) via pairwise ranking. It enables efficient architecture search using much fewer training examples. Moreover, we develop an architecture selection method to prune the search space and concentrate on more promising candidates. Extensive experiments on machine translation and language modeling tasks show that RankNAS can design high-performance architectures while being orders of magnitude faster than state-of-the-art NAS systems.