Entropy of Overcomplete Kernel Dictionaries

TL;DR

This paper introduces an entropy-based framework to evaluate overcomplete kernel dictionaries, linking diversity measures to information theory to assess atom distribution and spread in signal processing models.

Contribution

It develops a novel entropy-based approach to analyze overcomplete kernel dictionaries, deriving bounds related to diversity measures and examining entropy in input and feature spaces.

Findings

Lower bounds on entropy for various diversity measures

Analysis of entropy in input and feature spaces

Enhanced understanding of atom distribution in kernel dictionaries

Abstract

In signal analysis and synthesis, linear approximation theory considers a linear decomposition of any given signal in a set of atoms, collected into a so-called dictionary. Relevant sparse representations are obtained by relaxing the orthogonality condition of the atoms, yielding overcomplete dictionaries with an extended number of atoms. More generally than the linear decomposition, overcomplete kernel dictionaries provide an elegant nonlinear extension by defining the atoms through a mapping kernel function (e.g., the gaussian kernel). Models based on such kernel dictionaries are used in neural networks, gaussian processes and online learning with kernels. The quality of an overcomplete dictionary is evaluated with a diversity measure the distance, the approximation, the coherence and the Babel measures. In this paper, we develop a framework to examine overcomplete kernel dictionaries with the entropy from information theory. Indeed, a higher value of the entropy is associated to a further uniform spread of the atoms over the space. For each of the aforementioned diversity measures, we derive lower bounds on the entropy. Several definitions of the entropy are examined, with an extensive analysis in both the input space and the mapped feature space.