On the Value of ML Models

TL;DR

This paper emphasizes the importance of choosing evaluation metrics that reflect real-world value in ML models, especially in selective classification, and demonstrates how this approach offers practical insights.

Contribution

It advocates for evaluation metrics aligned with practical value and shows the significance of selective classification in understanding model quality.

Findings

Evaluation metrics should reflect practical value in ML models.

Selective classification provides meaningful insights into model quality.

Aligning metrics with real-world use cases improves model assessment.

Abstract

We argue that, when establishing and benchmarking Machine Learning (ML) models, the research community should favour evaluation metrics that better capture the value delivered by their model in practical applications. For a specific class of use cases -- selective classification -- we show that not only can it be simple enough to do, but that it has import consequences and provides insights what to look for in a ``good'' ML model.