Prayatul Matrix: A Direct Comparison Approach to Evaluate Performance of Supervised Machine Learning Models

TL;DR

This paper introduces Prayatul Matrix, a novel method for directly comparing supervised machine learning models at the individual instance level, providing deeper insights than traditional aggregate scores.

Contribution

The paper proposes the Prayatul Matrix and five new performance measures for instance-level comparison of ML models, enhancing evaluation granularity.

Findings

Prayatul Matrix offers more detailed model comparison insights.

New measures outperform traditional scores in revealing model effectiveness.

Method validated on diverse datasets and deep learning models.

Abstract

Performance comparison of supervised machine learning (ML) models are widely done in terms of different confusion matrix based scores obtained on test datasets. However, a dataset comprises several instances having different difficulty levels. Therefore, it is more logical to compare effectiveness of ML models on individual instances instead of comparing scores obtained for the entire dataset. In this paper, an alternative approach is proposed for direct comparison of supervised ML models in terms of individual instances within the dataset. A direct comparison matrix called \emph{Prayatul Matrix} is introduced, which accounts for comparative outcome of two ML algorithms on different instances of a dataset. Five different performance measures are designed based on prayatul matrix. Efficacy of the proposed approach as well as designed measures is analyzed with four classification techniques on three datasets. Also analyzed on four large-scale complex image datasets with four deep learning models namely ResNet50V2, MobileNetV2, EfficientNet, and XceptionNet. Results are evident that the newly designed measure are capable of giving more insight about the comparing ML algorithms, which were impossible with existing confusion matrix based scores like accuracy, precision and recall.