Analysis of the Relative Entropy Asymmetry in the Regularization of Empirical Risk Minimization

TL;DR

This paper investigates the asymmetry of relative entropy in empirical risk minimization with regularization, introducing a new Type-II regularization that extends the solution support and analyzing its properties and implications.

Contribution

It introduces Type-II regularization for ERM-RER, providing analytical solutions and insights into the support and bias effects of relative entropy as a regularizer.

Findings

Type-II regularization extends solution support beyond the reference measure.

Relative entropy causes the solution support to collapse into the reference measure support.

Closed-form expressions relate empirical risk to regularization parameters.

Abstract

The effect of the relative entropy asymmetry is analyzed in the empirical risk minimization with relative entropy regularization (ERM-RER) problem. A novel regularization is introduced, coined Type-II regularization, that allows for solutions to the ERM-RER problem with a support that extends outside the support of the reference measure. The solution to the new ERM-RER Type-II problem is analytically characterized in terms of the Radon-Nikodym derivative of the reference measure with respect to the solution. The analysis of the solution unveils the following properties of relative entropy when it acts as a regularizer in the ERM-RER problem: i) relative entropy forces the support of the Type-II solution to collapse into the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; ii) Type-II regularization is equivalent to classical relative entropy regularization with an appropriate transformation of the empirical risk function. Closed-form expressions of the expected empirical risk as a function of the regularization parameters are provided.