Empirical Risk Minimization with $f$-Divergence Regularization

TL;DR

This paper introduces a new framework for empirical risk minimization with $f$-divergence regularization, providing theoretical insights, a novel normalization function characterized by an ODE, and a practical algorithm for different divergence choices.

Contribution

It extends ERM-$f$DR applicability to more $f$-divergences, introduces the normalization function with an ODE characterization, and develops a numerical algorithm for practical computation.

Findings

The normalization function is characterized as a nonlinear ODE.

The proposed algorithm effectively computes risks under various $f$-divergences.

Structural equivalences between different $f$-divergence ERM problems are established.

Abstract

In this paper, the solution to the empirical risk minimization problem with $f$-divergence regularization (ERM-$f$DR) is presented and conditions under which the solution also serves as the solution to the minimization of the expected empirical risk subject to an $f$-divergence constraint are established. The proposed approach extends applicability to a broader class of $f$-divergences than previously reported and yields theoretical results that recover previously known results. Additionally, the difference between the expected empirical risk of the ERM-$f$DR solution and that of its reference measure is characterized, providing insights into previously studied cases of $f$-divergences. A central contribution is the introduction of the normalization function, a mathematical object that is critical in both the dual formulation and practical computation of the ERM-$f$DR solution. This work presents an implicit characterization of the normalization function as a nonlinear ordinary differential equation (ODE), establishes its key properties, and subsequently leverages them to construct a numerical algorithm for approximating the normalization factor under mild assumptions. Further analysis demonstrates structural equivalences between ERM-$f$DR problems with different $f$-divergences via transformations of the empirical risk. Finally, the proposed algorithm is used to compute the training and test risks of ERM-$f$DR solutions under different $f$-divergence regularizers. This numerical example highlights the practical implications of choosing different functions $f$ in ERM-$f$DR problems.