Variable transformations in consistent loss functions

TL;DR

This paper develops a theoretical framework to understand variable transformations in consistent loss functions, extending existing principles and enabling new applications in statistical and machine learning models.

Contribution

It provides formal characterizations of transformed loss functions and identification functions, broadening the theoretical foundation for designing loss functions in complex predictive tasks.

Findings

Framework applies to Bregman and expectile loss functions.

Constructs new identifiable and elicitable functionals.

Demonstrates practical utility with real-world data.

Abstract

The empirical use of variable transformations within (strictly) consistent loss functions is widespread, yet a theoretical understanding is lacking. To address this gap, we develop a theoretical framework that establishes formal characterizations of (strict) consistency for such transformed loss functions. Our analysis focuses on two interrelated cases: (a) transformations applied solely to the realization variable and (b) bijective transformations applied jointly to both the realization and prediction variables. These cases extend the well-established framework of transformations applied exclusively to the prediction variable, as formalized by Osband's revelation principle. We further develop analogous characterizations for (strict) identification functions. The resulting theoretical framework is broadly applicable to statistical and machine learning methodologies. For instance, we apply the framework to Bregman and expectile loss functions to interpret empirical findings from models trained with transformed loss functions and systematically construct new identifiable and elicitable functionals, which we term respectively $g$-transformed expectation and $g$-transformed expectile. Applications of the framework to simulated and real-world data illustrate its practical utility in diverse settings. By unifying theoretical insights with practical applications, this work advances principled methodologies for designing loss functions in complex predictive tasks.