A survey on multi-objective hyperparameter optimization algorithms for Machine Learning

TL;DR

This survey reviews multi-objective hyperparameter optimization algorithms in machine learning, highlighting recent developments, methodologies, and evaluation metrics from 2014 to 2020.

Contribution

It provides a comprehensive classification and comparison of multi-objective HPO algorithms, and discusses evaluation metrics and future research directions.

Findings

Classification of algorithms into metaheuristic and metamodel-based approaches

Analysis of quality metrics used for comparing multi-objective HPO methods

Identification of research gaps and future directions in multi-objective HPO

Abstract

Hyperparameter optimization (HPO) is a necessary step to ensure the best possible performance of Machine Learning (ML) algorithms. Several methods have been developed to perform HPO; most of these are focused on optimizing one performance measure (usually an error-based measure), and the literature on such single-objective HPO problems is vast. Recently, though, algorithms have appeared that focus on optimizing multiple conflicting objectives simultaneously. This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms, distinguishing between metaheuristic-based algorithms, metamodel-based algorithms, and approaches using a mixture of both. We also discuss the quality metrics used to compare multi-objective HPO procedures and present future research directions.