Learning Theory Approach to Minimum Error Entropy Criterion

TL;DR

This paper develops a learning theory framework for the minimum error entropy (MEE) criterion in regression, providing explicit error bounds and asymptotic analysis to understand its generalization capabilities.

Contribution

It introduces a novel theoretical analysis for MEE, including explicit error bounds and asymptotic behavior, addressing technical challenges unique to MEE compared to least squares methods.

Findings

Explicit error bounds in terms of hypothesis space capacity.

Asymptotic analysis of generalization error with Renyi's entropy.

Introduction of a semi-norm related to ranking algorithms.

Abstract

We consider the minimum error entropy (MEE) criterion and an empirical risk minimization learning algorithm in a regression setting. A learning theory approach is presented for this MEE algorithm and explicit error bounds are provided in terms of the approximation ability and capacity of the involved hypothesis space when the MEE scaling parameter is large. Novel asymptotic analysis is conducted for the generalization error associated with Renyi's entropy and a Parzen window function, to overcome technical difficulties arisen from the essential differences between the classical least squares problems and the MEE setting. A semi-norm and the involved symmetrized least squares error are introduced, which is related to some ranking algorithms.