Lift Up and Act! Classifier Performance in Resource-Constrained Applications

TL;DR

This paper explores how traditional classification metrics may not align with real-world resource-constrained decision-making scenarios, proposing the use of gains and lift to better evaluate classifier performance.

Contribution

It introduces resource-constrained classifier performance as a distinct task and discusses how gains and lift influence algorithm selection and outcomes.

Findings

Gains and lift provide more relevant performance measures in resource-limited contexts.

Traditional metrics like ROC and AUC may not reflect practical decision outcomes.

Class distribution impacts the effectiveness of different evaluation metrics.

Abstract

Classification tasks are common across many fields and applications where the decision maker's action is limited by resource constraints. In direct marketing only a subset of customers is contacted; scarce human resources limit the number of interviews to the most promising job candidates; limited donated organs are prioritized to those with best fit. In such scenarios, performance measures such as the classification matrix, ROC analysis, and even ranking metrics such as AUC measures outcomes different from the action of interest. At the same time, gains and lift that do measure the relevant outcome are rarely used by machine learners. In this paper we define resource-constrained classifier performance as a task distinguished from classification and ranking. We explain how gains and lift can lead to different algorithm choices and discuss the effect of class distribution.