Approximate Neural Architecture Search via Operation Distribution Learning

TL;DR

This paper introduces a stochastic approach to neural architecture search by focusing on the optimal operation distribution, which simplifies the search process and maintains high performance across various datasets and algorithms.

Contribution

It proposes a novel method that searches for operation distributions instead of specific architectures, enabling faster and more flexible NAS with comparable results.

Findings

Operation distribution effectively predicts architecture performance

The method accelerates NAS with minimal performance loss

Applicable across multiple NAS algorithms and datasets

Abstract

The standard paradigm in Neural Architecture Search (NAS) is to search for a fully deterministic architecture with specific operations and connections. In this work, we instead propose to search for the optimal operation distribution, thus providing a stochastic and approximate solution, which can be used to sample architectures of arbitrary length. We propose and show, that given an architectural cell, its performance largely depends on the ratio of used operations, rather than any specific connection pattern in typical search spaces; that is, small changes in the ordering of the operations are often irrelevant. This intuition is orthogonal to any specific search strategy and can be applied to a diverse set of NAS algorithms. Through extensive validation on 4 data-sets and 4 NAS techniques (Bayesian optimisation, differentiable search, local search and random search), we show that the operation distribution (1) holds enough discriminating power to reliably identify a solution and (2) is significantly easier to optimise than traditional encodings, leading to large speed-ups at little to no cost in performance. Indeed, this simple intuition significantly reduces the cost of current approaches and potentially enable NAS to be used in a broader range of applications.