Analyzing Cost-Sensitive Surrogate Losses via $\mathcal{H}$-calibration

TL;DR

This paper investigates whether cost-sensitive surrogate losses outperform cost-agnostic ones in training machine learning models, demonstrating their advantages both theoretically under certain assumptions and empirically on real datasets.

Contribution

It provides a theoretical analysis of $ ext{H}$-calibration showing when cost-sensitive surrogates are superior and empirically confirms their effectiveness on UCI datasets.

Findings

Cost-sensitive surrogates can outperform cost-agnostic ones for small models.

Theoretical results hold under common distributional assumptions.

Empirical results favor cost-sensitive surrogates on real datasets.

Abstract

This paper aims to understand whether machine learning models should be trained using cost-sensitive surrogates or cost-agnostic ones (e.g., cross-entropy). Analyzing this question through the lens of $\mathcal{H}$-calibration, we find that cost-sensitive surrogates can strictly outperform their cost-agnostic counterparts when learning small models under common distributional assumptions. Since these distributional assumptions are hard to verify in practice, we also show that cost-sensitive surrogates consistently outperform cost-agnostic surrogates on classification datasets from the UCI repository. Together, these make a strong case for using cost-sensitive surrogates in practice.