Default Machine Learning Hyperparameters Do Not Provide Informative Initialization for Bayesian Optimization

TL;DR

This study rigorously tests whether using default hyperparameters as initial points in Bayesian Optimization improves tuning efficiency, finding no significant advantage over random initialization across multiple models and datasets.

Contribution

It provides empirical evidence that default hyperparameters do not offer informative starting points for Bayesian Optimization, challenging common assumptions.

Findings

Default-informed initialization shows no significant performance gain.

Tighter priors around defaults only benefit early evaluations, not final results.

Practitioners should prioritize data-driven tuning over default hyperparameter reliance.

Abstract

Bayesian Optimization (BO) is a standard tool for hyperparameter tuning thanks to its sample efficiency on expensive black-box functions. While most BO pipelines begin with uniform random initialization, default hyperparameter values shipped with popular ML libraries such as scikit-learn encode implicit expert knowledge and could serve as informative starting points that accelerate convergence. This hypothesis, despite its intuitive appeal, has remained largely unexamined. We formalize the idea by initializing BO with points drawn from truncated Gaussian distributions centered at library defaults and compare the resulting trajectories against a uniform-random baseline. We conduct an extensive empirical evaluation spanning three BO back-ends (BoTorch, Optuna, Scikit-Optimize), three model families (Random Forests, Support Vector Machines, Multilayer Perceptrons), and five benchmark datasets covering classification and regression tasks. Performance is assessed through convergence speed and final predictive quality, and statistical significance is determined via one-sided binomial tests. Across all conditions, default-informed initialization yields no statistically significant advantage over purely random sampling, with p-values ranging from 0.141 to 0.908. A sensitivity analysis on the prior variance confirms that, while tighter concentration around the defaults improves early evaluations, this transient benefit vanishes as optimization progresses, leaving final performance unchanged. Our results provide no evidence that default hyperparameters encode useful directional information for optimization. We therefore recommend that practitioners treat hyperparameter tuning as an integral part of model development and favor principled, data-driven search strategies over heuristic reliance on library defaults.