Is One Hyperparameter Optimizer Enough?

TL;DR

This paper evaluates various hyperparameter optimizers for defect prediction in software analytics, revealing no single best method and that optimization often offers limited improvements over default settings.

Contribution

It systematically compares multiple hyperparameter tuning methods in software defect prediction, highlighting the nuanced effectiveness of different optimizers.

Findings

No optimizer was consistently best across measures.

Hyperparameter tuning often did not outperform default configurations.

Effectiveness of optimizers varies depending on the dataset and measure.

Abstract

Hyperparameter tuning is the black art of automatically finding a good combination of control parameters for a data miner. While widely applied in empirical Software Engineering, there has not been much discussion on which hyperparameter tuner is best for software analytics. To address this gap in the literature, this paper applied a range of hyperparameter optimizers (grid search, random search, differential evolution, and Bayesian optimization) to defect prediction problem. Surprisingly, no hyperparameter optimizer was observed to be `best' and, for one of the two evaluation measures studied here (F-measure), hyperparameter optimization, in 50\% cases, was no better than using default configurations. We conclude that hyperparameter optimization is more nuanced than previously believed. While such optimization can certainly lead to large improvements in the performance of classifiers used in software analytics, it remains to be seen which specific optimizers should be applied to a new dataset.