Model-based Asynchronous Hyperparameter and Neural Architecture Search

TL;DR

This paper presents a novel model-based asynchronous hyperparameter and neural architecture search method that leverages probabilistic modeling to efficiently optimize across multiple resource levels, demonstrating significant speed-ups on various benchmarks.

Contribution

The paper introduces a new asynchronous multi-fidelity search method combining Gaussian process models with Hyperband, enabling more efficient neural architecture and hyperparameter optimization.

Findings

Substantial speed-ups over state-of-the-art methods.

Effective across tabular, image, and language benchmarks.

Distributed implementation with open-source release.

Abstract

We introduce a model-based asynchronous multi-fidelity method for hyperparameter and neural architecture search that combines the strengths of asynchronous Hyperband and Gaussian process-based Bayesian optimization. At the heart of our method is a probabilistic model that can simultaneously reason across hyperparameters and resource levels, and supports decision-making in the presence of pending evaluations. We demonstrate the effectiveness of our method on a wide range of challenging benchmarks, for tabular data, image classification and language modelling, and report substantial speed-ups over current state-of-the-art methods. Our new methods, along with asynchronous baselines, are implemented in a distributed framework which will be open sourced along with this publication.