Partial VOROS: A Cost-aware Performance Metric for Binary Classifiers with Precision and Capacity Constraints

TL;DR

This paper introduces partial VOROS, a new cost-aware performance metric for binary classifiers that accounts for precision and capacity constraints, improving classifier ranking in sensitive applications.

Contribution

The paper proposes a novel geometric and metric framework, partial VOROS, to evaluate classifiers under real-world constraints like precision and capacity.

Findings

Partial VOROS effectively ranks classifiers in hospital alert systems.

The feasible region in ROC space is polygon-shaped, aiding metric computation.

Experiments show partial VOROS outperforms existing metrics in mortality risk prediction.

Abstract

The ROC curve is widely used to assess binary classifiers. Yet for some applications, such as alert systems for monitoring hospitalized patients, conventional ROC analysis cannot meet two key deployment needs: enforcing a constraint on precision to avoid false alarm fatigue and imposing an upper bound on the number of predicted positives to represent the capacity of hospital staff. The usual area under the curve metric also does not reflect asymmetric costs for false positives and false negatives. In this paper we address all three of these issues. First, we show how the subset of classifiers that meet precision and capacity constraints occupy a feasible region in ROC space. We establish the polygon-shaped geometry of this region. We then define the partial area of lesser classifiers, a performance metric that is monotonic with cost and only accounts for the feasible region. Averaging this area over a desired distribution for cost parameters results in the partial volume over the ROC surface, or partial VOROS. In experiments predicting mortality risk from vital sign history on several datasets, we show this cost-aware metric can outperform alternatives at ranking classifiers for in-hospital alerts.