Developing a Dataset-Adaptive, Normalized Metric for Machine Learning Model Assessment: Integrating Size, Complexity, and Class Imbalance

TL;DR

This paper introduces a dataset-adaptive, normalized metric that considers dataset size, complexity, and class imbalance to improve model evaluation, especially on challenging, limited, or high-dimensional datasets.

Contribution

The study presents a novel, scalable metric that integrates dataset characteristics for more accurate model assessment across various machine learning tasks.

Findings

The metric accurately predicts model performance in small, unbalanced datasets.

Experimental validation shows improved evaluation consistency across tasks.

The approach aids in resource allocation and model optimization in ML workflows.

Abstract

Traditional metrics like accuracy, F1-score, and precision are frequently used to evaluate machine learning models, however they may not be sufficient for evaluating performance on tiny, unbalanced, or high-dimensional datasets. A dataset-adaptive, normalized metric that incorporates dataset characteristics like size, feature dimensionality, class imbalance, and signal-to-noise ratio is presented in this study. Early insights into the model's performance potential in challenging circumstances are provided by the suggested metric, which offers a scalable and adaptable evaluation framework. The metric's capacity to accurately forecast model scalability and performance is demonstrated via experimental validation spanning classification, regression, and clustering tasks, guaranteeing solid assessments in settings with limited data. This method has important ramifications for effective resource allocation and model optimization in machine learning workflows.