Beyond Surrogates: A Quantitative Analysis for Inter-Metric Relationships

TL;DR

This paper introduces a unified theoretical framework to analyze and quantify the relationships between different evaluation metrics, addressing the common disconnect between offline validation gains and online performance in industrial applications.

Contribution

It proposes a novel framework for inter-metric relationship analysis, categorizes metrics, and explores structural asymmetries to improve alignment between offline metrics and online outcomes.

Findings

Identifies structural asymmetries in regret transfer between metrics

Provides a theoretical basis for designing aligned evaluation systems

Bridges the gap between offline metrics and online performance

Abstract

The Consistency property between surrogate losses and evaluation metrics has been extensively studied to ensure that minimizing a loss leads to metric optimality. However, the direct relationship between different evaluation metrics remains significantly underexplored. This theoretical gap results in the "Metric Mismatch" frequently observed in industrial applications, where gains in offline validation metrics fail to translate into online performance. To bridge this disconnection, this paper proposes a unified theoretical framework designed to quantify the relationships between metrics. We categorize metrics into different classes to facilitate a comparative analysis across different mathematical forms and interrogates these relationships through Bayes-Optimal Set and Regret Transfer. Through this framework, we provide a new perspective on identifying the structural asymmetry in regret transfer, enabling the design of evaluation systems that are theoretically guaranteed to align offline improvements with online objectives.