The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks

TL;DR

This study systematically evaluates 12 feature scaling techniques across 14 machine learning algorithms and 16 datasets, revealing how scaling impacts performance and computational costs, with insights tailored to specific models.

Contribution

It provides a comprehensive empirical analysis of feature scaling effects on diverse models and datasets, offering practical guidance for selecting optimal scaling methods.

Findings

Ensemble methods are largely insensitive to feature scaling.

Logistic Regression, SVMs, TabNet, and MLPs show performance heavily influenced by scaling.

Scaling choice significantly affects model accuracy and computational efficiency.

Abstract

This research addresses the critical lack of comprehensive studies on feature scaling by systematically evaluating 12 scaling techniques - including several less common transformations - across 14 different Machine Learning algorithms and 16 datasets for classification and regression tasks. We meticulously analyzed impacts on predictive performance (using metrics such as accuracy, MAE, MSE, and $R^2$) and computational costs (training time, inference time, and memory usage). Key findings reveal that while ensemble methods (such as Random Forest and gradient boosting models like XGBoost, CatBoost and LightGBM) demonstrate robust performance largely independent of scaling, other widely used models such as Logistic Regression, SVMs, TabNet, and MLPs show significant performance variations highly dependent on the chosen scaler. This extensive empirical analysis, with all source code, experimental results, and model parameters made publicly available to ensure complete transparency and reproducibility, offers model-specific crucial guidance to practitioners on the need for an optimal selection of feature scaling techniques.