Optimized Kernel Entropy Components

TL;DR

This paper introduces OKECA, an extension of KECA that optimizes feature extraction based on entropy, resulting in more expressive features and improved robustness in density estimation.

Contribution

The paper proposes OKECA, which enhances KECA by optimizing an extra rotation for maximum entropy preservation, leading to more efficient feature extraction.

Findings

OKECA produces features with higher expressive power than KECA.

Maximum likelihood is the best criterion for kernel parameter estimation.

OKECA is more robust to kernel length-scale selection in density estimation.

Abstract

This work addresses two main issues of the standard Kernel Entropy Component Analysis (KECA) algorithm: the optimization of the kernel decomposition and the optimization of the Gaussian kernel parameter. KECA roughly reduces to a sorting of the importance of kernel eigenvectors by entropy instead of by variance as in Kernel Principal Components Analysis. In this work, we propose an extension of the KECA method, named Optimized KECA (OKECA), that directly extracts the optimal features retaining most of the data entropy by means of compacting the information in very few features (often in just one or two). The proposed method produces features which have higher expressive power. In particular, it is based on the Independent Component Analysis (ICA) framework, and introduces an extra rotation to the eigen-decomposition, which is optimized via gradient ascent search. This maximum entropy preservation suggests that OKECA features are more efficient than KECA features for density estimation. In addition, a critical issue in both methods is the selection of the kernel parameter since it critically affects the resulting performance. Here we analyze the most common kernel length-scale selection criteria. Results of both methods are illustrated in different synthetic and real problems. Results show that 1) OKECA returns projections with more expressive power than KECA, 2) the most successful rule for estimating the kernel parameter is based on maximum likelihood, and 3) OKECA is more robust to the selection of the length-scale parameter in kernel density estimation.