Importance of Tuning Hyperparameters of Machine Learning Algorithms

TL;DR

This paper investigates the importance of hyperparameter tuning in machine learning, proposing a methodology to assess when defaults suffice and demonstrating that tuning is not always necessary for optimal performance.

Contribution

It introduces a non-inferiority testing approach to evaluate hyperparameter importance and provides a procedure to identify reasonable default values, validated on extensive benchmark datasets.

Findings

Some hyperparameters can be set to defaults without performance loss

Default hyperparameters sometimes outperform limited search tuning

Methodology helps identify when tuning is unnecessary

Abstract

The performance of many machine learning algorithms depends on their hyperparameter settings. The goal of this study is to determine whether it is important to tune a hyperparameter or whether it can be safely set to a default value. We present a methodology to determine the importance of tuning a hyperparameter based on a non-inferiority test and tuning risk: the performance loss that is incurred when a hyperparameter is not tuned, but set to a default value. Because our methods require the notion of a default parameter, we present a simple procedure that can be used to determine reasonable default parameters. We apply our methods in a benchmark study using 59 datasets from OpenML. Our results show that leaving particular hyperparameters at their default value is non-inferior to tuning these hyperparameters. In some cases, leaving the hyperparameter at its default value even outperforms tuning it using a search procedure with a limited number of iterations.