A graphical analysis of cost-sensitive regression problems

TL;DR

This paper introduces a novel graphical framework called RROC space for analyzing regression models under cost-sensitive conditions, revealing that the area over the curve correlates with error variance.

Contribution

It proposes the RROC space representation for regression, extending ROC analysis beyond classification, and establishes a key link between AOC and error variance.

Findings

RROC curves can be generated by adjusting a shift parameter.

The area over the RROC curve (AOC) equals the error variance times a constant.

The RROC framework allows graphical evaluation of cost-sensitive regression models.

Abstract

Several efforts have been done to bring ROC analysis beyond (binary) classification, especially in regression. However, the mapping and possibilities of these proposals do not correspond to what we expect from the analysis of operating conditions, dominance, hybrid methods, etc. In this paper we present a new representation of regression models in the so-called regression ROC (RROC) space. The basic idea is to represent over-estimation on the x axis and under-estimation on the y axis. The curves are just drawn by adjusting a shift, a constant that is added (or subtracted) to the predictions, and plays a similar role as a threshold in classification. From here, we develop the notions of optimal operating condition, convexity, dominance, and explore several evaluation metrics that can be shown graphically, such as the area over the RROC curve (AOC). In particular, we show a novel and significant result, the AOC is equal to the error variance (multiplied by a factor which does not depend on the model). The derivation of RROC curves with non-constant shifts and soft regression models, and the relation with cost plots is also discussed.