Hyperparameter Optimization in Machine Learning

TL;DR

This paper surveys methods for automating hyperparameter optimization in machine learning, emphasizing techniques, extensions, and future research directions to improve efficiency and effectiveness.

Contribution

It provides a comprehensive overview of hyperparameter optimization techniques, connecting various approaches and highlighting open challenges in the field.

Findings

Various hyperparameter optimization methods are compared and analyzed.

Extensions like online and multi-objective optimization are discussed.

Connections to meta-learning and neural architecture search are explored.

Abstract

Hyperparameters are configuration variables controlling the behavior of machine learning algorithms. They are ubiquitous in machine learning and artificial intelligence and the choice of their values determines the effectiveness of systems based on these technologies. Manual hyperparameter search is often time-consuming and becomes infeasible when the number of hyperparameters is large. Automating the search is an important step towards advancing, streamlining, and systematizing machine learning, freeing researchers and practitioners alike from the burden of finding a good set of hyperparameters by trial and error. In this survey, we present a unified treatment of hyperparameter optimization, providing the reader with examples, insights into the state-of-the-art, and numerous links to further reading. We cover the main families of techniques to automate hyperparameter search, often referred to as hyperparameter optimization or tuning, including random and quasi-random search, bandit-, model-, population-, and gradient-based approaches. We further discuss extensions, including online, constrained, and multi-objective formulations, touch upon connections with other fields, such as meta-learning and neural architecture search, and conclude with open questions and future research directions.