A Collection of Quality Diversity Optimization Problems Derived from Hyperparameter Optimization of Machine Learning Models

TL;DR

This paper introduces a new set of Quality Diversity Optimization problems focused on hyperparameter tuning of machine learning models, incorporating novel features like interpretability and resource efficiency, and evaluates different optimizers on these benchmarks.

Contribution

It proposes a novel benchmark suite for Quality Diversity Optimization in hyperparameter tuning, utilizing surrogate models for efficient evaluation and exploring new feature functions.

Findings

Different optimizers show varying performance on the new benchmarks

Surrogate models enable faster benchmarking of hyperparameter optimization methods

The study highlights future challenges and directions for applying Quality Diversity Optimization in machine learning

Abstract

The goal of Quality Diversity Optimization is to generate a collection of diverse yet high-performing solutions to a given problem at hand. Typical benchmark problems are, for example, finding a repertoire of robot arm configurations or a collection of game playing strategies. In this paper, we propose a set of Quality Diversity Optimization problems that tackle hyperparameter optimization of machine learning models - a so far underexplored application of Quality Diversity Optimization. Our benchmark problems involve novel feature functions, such as interpretability or resource usage of models. To allow for fast and efficient benchmarking, we build upon YAHPO Gym, a recently proposed open source benchmarking suite for hyperparameter optimization that makes use of high performing surrogate models and returns these surrogate model predictions instead of evaluating the true expensive black box function. We present results of an initial experimental study comparing different Quality Diversity optimizers on our benchmark problems. Furthermore, we discuss future directions and challenges of Quality Diversity Optimization in the context of hyperparameter optimization.