Generalization and Robustness of the Tilted Empirical Risk

TL;DR

This paper analyzes the generalization and robustness properties of the tilted empirical risk, especially under negative tilt, providing theoretical bounds and empirical validation for its performance in noisy and shifting data environments.

Contribution

It offers the first uniform and information-theoretic bounds on tilted generalization error under negative tilt with unbounded loss functions and explores robustness guarantees under distribution shift.

Findings

Bounds on tilted generalization error with convergence rate of O(n^{-rac{psilon}{1+psilon}})

Theoretical guarantees for robustness against noisy outliers and distribution shifts

Empirical validation of bounds and tilt selection in simple experimental setups

Abstract

The generalization error (risk) of a supervised statistical learning algorithm quantifies its prediction ability on previously unseen data. Inspired by exponential tilting, \citet{li2020tilted} proposed the {\it tilted empirical risk} (TER) as a non-linear risk metric for machine learning applications such as classification and regression problems. In this work, we examine the generalization error of the tilted empirical risk in the robustness regime under \textit{negative tilt}. Our first contribution is to provide uniform and information-theoretic bounds on the {\it tilted generalization error}, defined as the difference between the population risk and the tilted empirical risk, under negative tilt for unbounded loss function under bounded $(1+\epsilon)$-th moment of loss function for some $\epsilon\in(0,1]$ with a convergence rate of $O(n^{-\epsilon/(1+\epsilon)})$ where $n$ is the number of training samples, revealing a novel application for TER under no distribution shift. Secondly, we study the robustness of the tilted empirical risk with respect to noisy outliers at training time and provide theoretical guarantees under distribution shift for the tilted empirical risk. We empirically corroborate our findings in simple experimental setups where we evaluate our bounds to select the value of tilt in a data-driven manner.