Behavior of Hyper-Parameters for Selected Machine Learning Algorithms: An Empirical Investigation

TL;DR

This paper empirically investigates how hyper-parameters affect the performance of XGB, RF, and FFNN algorithms, providing insights and guidelines for more efficient hyper-parameter tuning.

Contribution

It offers a detailed analysis of hyper-parameter behavior across three ML algorithms and proposes guidelines to reduce tuning search space.

Findings

Hyper-parameters significantly influence model performance.

Certain hyper-parameters are more critical for specific algorithms.

Performance stability is observed near optimal hyper-parameter regions.

Abstract

Hyper-parameters (HPs) are an important part of machine learning (ML) model development and can greatly influence performance. This paper studies their behavior for three algorithms: Extreme Gradient Boosting (XGB), Random Forest (RF), and Feedforward Neural Network (FFNN) with structured data. Our empirical investigation examines the qualitative behavior of model performance as the HPs vary, quantifies the importance of each HP for different ML algorithms, and stability of the performance near the optimal region. Based on the findings, we propose a set of guidelines for efficient HP tuning by reducing the search space.