Asymptotic Properties of Matthews Correlation Coefficient

TL;DR

This paper develops and evaluates methods for constructing confidence intervals for the Matthews correlation coefficient, enhancing statistical inference and comparison of classifiers in imbalanced classification tasks.

Contribution

It introduces and assesses new asymptotic confidence interval methods for MCC, addressing a gap in statistical inference for this metric.

Findings

Methods perform well in simulations across scenarios

Confidence intervals provide reliable statistical inference

Real data analysis demonstrates practical utility

Abstract

Evaluating classifications is crucial in statistics and machine learning, as it influences decision-making across various fields, such as patient prognosis and therapy in critical conditions. The Matthews correlation coefficient (MCC) is recognized as a performance metric with high reliability, offering a balanced measurement even in the presence of class imbalances. Despite its importance, there remains a notable lack of comprehensive research on the statistical inference of MCC. This deficiency often leads to studies merely validating and comparing MCC point estimates, a practice that, while common, overlooks the statistical significance and reliability of results. Addressing this research gap, our paper introduces and evaluates several methods to construct asymptotic confidence intervals for the single MCC and the differences between MCCs in paired designs. Through simulations across various scenarios, we evaluate the finite-sample behavior of these methods and compare their performances. Furthermore, through real data analysis, we illustrate the potential utility of our findings in comparing binary classifiers, highlighting the possible contributions of our research in this field.