Visualization of Tradeoff in Evaluation: from Precision-Recall & PN to LIFT, ROC & BIRD

TL;DR

This paper reviews and extends visualization techniques for evaluating classification systems, focusing on trade-offs and multiclass scenarios, by analyzing traditional and new probabilistic and information-theoretic variants.

Contribution

It introduces new probabilistic and information-theoretic variants of LIFT to better visualize multiclass and unbalanced data evaluation trade-offs.

Findings

Analysis of dichotomous evaluation methods

Development of new LIFT variants for multiclass data

Enhanced visualization of trade-offs in classification evaluation

Abstract

Evaluation often aims to reduce the correctness or error characteristics of a system down to a single number, but that always involves trade-offs. Another way of dealing with this is to quote two numbers, such as Recall and Precision, or Sensitivity and Specificity. But it can also be useful to see more than this, and a graphical approach can explore sensitivity to cost, prevalence, bias, noise, parameters and hyper-parameters. Moreover, most techniques are implicitly based on two balanced classes, and our ability to visualize graphically is intrinsically two dimensional, but we often want to visualize in a multiclass context. We review the dichotomous approaches relating to Precision, Recall, and ROC as well as the related LIFT chart, exploring how they handle unbalanced and multiclass data, and deriving new probabilistic and information theoretic variants of LIFT that help deal with the issues associated with the handling of multiple and unbalanced classes.