Heed the Noise in Performance Evaluations in Neural Architecture Search

TL;DR

This paper highlights the impact of stochastic noise in neural architecture search evaluations and demonstrates that averaging over multiple runs improves the quality of found architectures, especially on small, variable datasets.

Contribution

It proposes a method to reduce noise in NAS by averaging over multiple training runs and evaluates its effectiveness across different search algorithms and datasets.

Findings

Reducing evaluation noise improves architecture quality.

Averaging over multiple runs is beneficial for small, variable datasets.

All considered search algorithms benefit from noise reduction.

Abstract

Neural Architecture Search (NAS) has recently become a topic of great interest. However, there is a potentially impactful issue within NAS that remains largely unrecognized: noise. Due to stochastic factors in neural network initialization, training, and the chosen train/validation dataset split, the performance evaluation of a neural network architecture, which is often based on a single learning run, is also stochastic. This may have a particularly large impact if a dataset is small. We therefore propose to reduce this noise by evaluating architectures based on average performance over multiple network training runs using different random seeds and cross-validation. We perform experiments for a combinatorial optimization formulation of NAS in which we vary noise reduction levels. We use the same computational budget for each noise level in terms of network training runs, i.e., we allow less architecture evaluations when averaging over more training runs. Multiple search algorithms are considered, including evolutionary algorithms which generally perform well for NAS. We use two publicly available datasets from the medical image segmentation domain where datasets are often limited and variability among samples is often high. Our results show that reducing noise in architecture evaluations enables finding better architectures by all considered search algorithms.