Hyper-Parameter Optimization: A Review of Algorithms and Applications

TL;DR

This review paper discusses the key hyper-parameters in neural network training, compares various optimization algorithms and tools for hyper-parameter optimization, and highlights challenges and solutions in applying HPO to deep learning.

Contribution

It provides a comprehensive overview of HPO algorithms, tools, and challenges, aiding researchers and practitioners in selecting suitable methods for deep learning models.

Findings

Major HPO algorithms vary in efficiency and accuracy for deep learning.

Existing tools support state-of-the-art algorithms and integrate with popular frameworks.

Challenges include computational resource limitations and evaluation difficulties.

Abstract

Since deep neural networks were developed, they have made huge contributions to everyday lives. Machine learning provides more rational advice than humans are capable of in almost every aspect of daily life. However, despite this achievement, the design and training of neural networks are still challenging and unpredictable procedures. To lower the technical thresholds for common users, automated hyper-parameter optimization (HPO) has become a popular topic in both academic and industrial areas. This paper provides a review of the most essential topics on HPO. The first section introduces the key hyper-parameters related to model training and structure, and discusses their importance and methods to define the value range. Then, the research focuses on major optimization algorithms and their applicability, covering their efficiency and accuracy especially for deep learning networks. This study next reviews major services and toolkits for HPO, comparing their support for state-of-the-art searching algorithms, feasibility with major deep learning frameworks, and extensibility for new modules designed by users. The paper concludes with problems that exist when HPO is applied to deep learning, a comparison between optimization algorithms, and prominent approaches for model evaluation with limited computational resources.