A Dual Optimization View to Empirical Risk Minimization with f-Divergence Regularization

TL;DR

This paper introduces a dual optimization framework for empirical risk minimization with f-divergence regularization, utilizing Legendre-Fenchel transform and implicit functions to derive a nonlinear ODE for efficient computation.

Contribution

It presents a novel dual formulation of ERM-fDR, connecting the solution to a normalization function via implicit functions and nonlinear ODEs, enabling efficient computation.

Findings

Derived a nonlinear ODE expression for the normalization function.

Provided a computationally efficient method for normalization function calculation.

Connected dual optimization to implicit functions and Legendre-Fenchel transform.

Abstract

The dual formulation of empirical risk minimization with f-divergence regularization (ERM-fDR) is introduced. The solution of the dual optimization problem to the ERM-fDR is connected to the notion of normalization function introduced as an implicit function. This dual approach leverages the Legendre-Fenchel transform and the implicit function theorem to provide a nonlinear ODE expression to the normalization function. Furthermore, the nonlinear ODE expression and its properties provide a computationally efficient method to calculate the normalization function of the ERM-fDR solution under a mild condition.