Multiclass ROC

TL;DR

This paper introduces a novel multi-class ROC evaluation method that addresses limitations of existing approaches by providing meaningful visualizations, handling imbalanced data, allowing misclassification costs, and quantifying uncertainty.

Contribution

We propose a new multi-class ROC evaluation metric based on pair-wise TPR and FPR, with visualization and confidence intervals, improving upon existing methods.

Findings

The new method effectively visualizes multi-class classifier performance.

It accommodates misclassification costs and provides confidence intervals.

Simulation studies show improved evaluation accuracy over traditional AUC methods.

Abstract

Model evaluation is of crucial importance in modern statistics application. The construction of ROC and calculation of AUC have been widely used for binary classification evaluation. Recent research generalizing the ROC/AUC analysis to multi-class classification has problems in at least one of the four areas: 1. failure to provide sensible plots 2. being sensitive to imbalanced data 3. unable to specify mis-classification cost and 4. unable to provide evaluation uncertainty quantification. Borrowing from a binomial matrix factorization model, we provide an evaluation metric summarizing the pair-wise multi-class True Positive Rate (TPR) and False Positive Rate (FPR) with one-dimensional vector representation. Visualization on the representation vector measures the relative speed of increment between TPR and FPR across all the classes pairs, which in turns provides a ROC plot for the multi-class counterpart. An integration over those factorized vector provides a binary AUC-equivalent summary on the classifier performance. Mis-clasification weights specification and bootstrapped confidence interval are also enabled to accommodate a variety of of evaluation criteria. To support our findings, we conducted extensive simulation studies and compared our method to the pair-wise averaged AUC statistics on benchmark datasets.