Instance-Level Costs for Nuanced Classifier Evaluation

TL;DR

This paper introduces NEC, a new metric for classifier evaluation that accounts for per-example costs, revealing that models often make mistakes on low-cost, ambiguous cases.

Contribution

The paper proposes normalized excess cost (NEC), a cost-sensitive evaluation metric, and analyzes its implications across various data types and training strategies.

Findings

NEC is often significantly lower than error rate, indicating errors mostly occur on low-cost ambiguous cases.

Incorporating costs into training yields inconsistent benefits, improving mainly when costs are predictable from inputs.

Models with 5% error rate can achieve 1.8% NEC, showing the importance of cost-aware evaluation.

Abstract

Standard classification treats all errors equally, but in content moderation, medical screening, and safety-critical applications, mistakes on clear-cut cases are far more costly than errors on ambiguous ones. We propose normalized excess cost (NEC), a metric that weights classification errors by per-example costs and reduces to standard error rate when costs are uniform. Costs can derive from annotator vote margins, distance from decision thresholds, or confidence ratings. Across text, image, and tabular benchmarks, we find that NEC is often substantially lower than error rate -- models with 5\% error rate can achieve 1.8\% NEC -- revealing that most mistakes concentrate on ambiguous, low-cost examples. However, incorporating costs into training via loss weighting, sampling strategies, or regression yields inconsistent benefits: improvements appear only when costs are predictable from input features, as in our synthetic control, while real-world datasets show mixed or negligible gains. Our framework provides a practical methodology for deriving and evaluating instance-level misclassification costs, even when cost-sensitive training offers limited benefit.