Kernel Mean Shrinkage Estimators

TL;DR

This paper introduces kernel mean shrinkage estimators (KMSEs) that improve the estimation of kernel means in RKHSs by leveraging the Stein phenomenon, showing superior performance especially in high-dimensional, small-sample settings.

Contribution

The paper proposes a novel family of kernel mean shrinkage estimators that are theoretically justified and empirically outperform the standard empirical mean estimator.

Findings

KMSEs outperform the standard estimator in high-dimensional, small-sample scenarios.

Theoretical analysis supports the effectiveness of the shrinkage approach.

Empirical results demonstrate improved accuracy of kernel mean estimation.

Abstract

A mean function in a reproducing kernel Hilbert space (RKHS), or a kernel mean, is central to kernel methods in that it is used by many classical algorithms such as kernel principal component analysis, and it also forms the core inference step of modern kernel methods that rely on embedding probability distributions in RKHSs. Given a finite sample, an empirical average has been used commonly as a standard estimator of the true kernel mean. Despite a widespread use of this estimator, we show that it can be improved thanks to the well-known Stein phenomenon. We propose a new family of estimators called kernel mean shrinkage estimators (KMSEs), which benefit from both theoretical justifications and good empirical performance. The results demonstrate that the proposed estimators outperform the standard one, especially in a "large d, small n" paradigm.