Generalization Error of $f$-Divergence Stabilized Algorithms via Duality

TL;DR

This paper introduces a dual formulation for $f$-divergence regularized algorithms, enabling explicit analysis of their generalization error and extending solutions to constrained optimization problems.

Contribution

It develops a dual approach using Legendre-Fenchel transform for $f$-divergence regularized algorithms, providing explicit generalization error bounds and extending to constrained problems.

Findings

Dual formulation simplifies computation of ERM-$f$DR solutions.

Explicit generalization error bounds derived for algorithms.

Extension of solutions to constrained optimization problems.

Abstract

The solution to empirical risk minimization with $f$-divergence regularization (ERM-$f$DR) is extended to constrained optimization problems, establishing conditions for equivalence between the solution and constraints. A dual formulation of ERM-$f$DR is introduced, providing a computationally efficient method to derive the normalization function of the ERM-$f$DR solution. This dual approach leverages the Legendre-Fenchel transform and the implicit function theorem, enabling explicit characterizations of the generalization error for general algorithms under mild conditions, and another for ERM-$f$DR solutions.