Tunability: Importance of Hyperparameters of Machine Learning Algorithms

TL;DR

This paper formalizes the concept of hyperparameter tunability in machine learning, introduces measures for assessing it, and provides empirical benchmarks across multiple datasets and algorithms to guide tuning strategies.

Contribution

It offers a statistical formalization of hyperparameter tuning, proposes measures for tunability, and provides empirical benchmarks to inform tuning decisions.

Findings

Default hyperparameter values identified for common algorithms

Guidelines on when tuning is beneficial based on tunability measures

Benchmark results across 38 datasets showing hyperparameter importance

Abstract

Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them. Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure. The goal of this paper is two-fold. Firstly, we formalize the problem of tuning from a statistical point of view, define data-based defaults and suggest general measures quantifying the tunability of hyperparameters of algorithms. Secondly, we conduct a large-scale benchmarking study based on 38 datasets from the OpenML platform and six common machine learning algorithms. We apply our measures to assess the tunability of their parameters. Our results yield default values for hyperparameters and enable users to decide whether it is worth conducting a possibly time consuming tuning strategy, to focus on the most important hyperparameters and to chose adequate hyperparameter spaces for tuning.