On Rank Graduation Metrics for High Dimensional Ordinal Data

TL;DR

This paper introduces a new family of Rank Graduation metrics for evaluating machine learning classification accuracy on high-dimensional ordinal data, addressing the challenge of lacking natural metrics.

Contribution

It develops a mathematical framework for comparing ordinal data using RGX metrics, linking them to existing metrics and demonstrating their effectiveness through extensive experiments.

Findings

RGX metrics effectively quantify explained variability in ordinal data

They demonstrate robustness and interpretability across diverse datasets

RGX metrics outperform traditional measures in certain classification scenarios

Abstract

Evaluating the reliability of machine learning classifications remains a fundamental challenge in Artificial Intelligence (AI), particularly when the target variable is multidimensional. Classification variables can be expressed by means of a categorical scale which, at best, is ordinal. Because ordinal data lack a natural metric structure in their underlying space, most conventional distance measures aimed at assessing the accuracy of machine learning classifications cannot be directly or meaningfully applied. In this paper, we develop a mathematical framework for comparing ordinal data based on a family of Rank Graduation $(\mathrm{RGX}_p)$ \emph{metrics}. We demonstrate that these metrics can quantify the proportion of variability of the response explained by the predictions, in a similar manner as the predictive $R^2$ for continuous response variables. After establishing theoretical connections between the $\mathrm{RGX}_p$ family and other prominent metrics in AI, we conduct extensive experiments across diverse datasets and learning tasks to evaluate their empirical performance. The results underscore the versatility, interpretability, and robustness of the $\mathrm{RGX}_p$ metrics as a principled foundation for developing trustworthy and SAFE AI systems.