Comparing Cluster-Based Cross-Validation Strategies for Machine Learning Model Evaluation

TL;DR

This paper evaluates cluster-based cross-validation strategies for machine learning, proposing a new method combining Mini Batch K-Means with class stratification, and compares their effectiveness across diverse datasets and algorithms.

Contribution

It introduces a novel cross-validation technique using Mini Batch K-Means with class stratification and provides an extensive experimental comparison of clustering algorithms for data splitting.

Findings

Mini Batch K-Means with class stratification reduces bias and variance on balanced datasets.

Traditional stratified cross-validation performs best on imbalanced datasets.

No clustering algorithm consistently outperforms others across all scenarios.

Abstract

Cross-validation plays a fundamental role in Machine Learning, enabling robust evaluation of model performance and preventing overestimation on training and validation data. However, one of its drawbacks is the potential to create data subsets (folds) that do not adequately represent the diversity of the original dataset, which can lead to biased performance estimates. The objective of this work is to deepen the investigation of cluster-based cross-validation strategies by analyzing the performance of different clustering algorithms through experimental comparison. Additionally, a new cross-validation technique that combines Mini Batch K-Means with class stratification is proposed. Experiments were conducted on 20 datasets (both balanced and imbalanced) using four supervised learning algorithms, comparing cross-validation strategies in terms of bias, variance, and computational cost. The technique that uses Mini Batch K-Means with class stratification outperformed others in terms of bias and variance on balanced datasets, though it did not significantly reduce computational cost. On imbalanced datasets, traditional stratified cross-validation consistently performed better, showing lower bias, variance, and computational cost, making it a safe choice for performance evaluation in scenarios with class imbalance. In the comparison of different clustering algorithms, no single algorithm consistently stood out as superior. Overall, this work contributes to improving predictive model evaluation strategies by providing a deeper understanding of the potential of cluster-based data splitting techniques and reaffirming the effectiveness of well-established strategies like stratified cross-validation. Moreover, it highlights perspectives for increasing the robustness and reliability of model evaluations, especially in datasets with clustering characteristics.