The choice of scaling technique matters for classification performance

TL;DR

This study systematically compares five data scaling techniques across twenty classification algorithms and eighty-two datasets, revealing that the choice of scaling significantly impacts model performance and can sometimes be more critical than not scaling at all.

Contribution

It provides a comprehensive experimental analysis of how different scaling methods affect classification performance, highlighting the importance of selecting appropriate scaling techniques.

Findings

The choice of scaling technique significantly affects classification accuracy.

Inadequate scaling can harm performance more than no scaling.

Ensemble models' sensitivity to scaling mirrors that of their base models.

Abstract

Dataset scaling, also known as normalization, is an essential preprocessing step in a machine learning pipeline. It is aimed at adjusting attributes scales in a way that they all vary within the same range. This transformation is known to improve the performance of classification models, but there are several scaling techniques to choose from, and this choice is not generally done carefully. In this paper, we execute a broad experiment comparing the impact of 5 scaling techniques on the performances of 20 classification algorithms among monolithic and ensemble models, applying them to 82 publicly available datasets with varying imbalance ratios. Results show that the choice of scaling technique matters for classification performance, and the performance difference between the best and the worst scaling technique is relevant and statistically significant in most cases. They also indicate that choosing an inadequate technique can be more detrimental to classification performance than not scaling the data at all. We also show how the performance variation of an ensemble model, considering different scaling techniques, tends to be dictated by that of its base model. Finally, we discuss the relationship between a model's sensitivity to the choice of scaling technique and its performance and provide insights into its applicability on different model deployment scenarios. Full results and source code for the experiments in this paper are available in a GitHub repository.\footnote{https://github.com/amorimlb/scaling\_matters}