Compressed Empirical Measures (in finite dimensions)

TL;DR

This paper investigates methods for compressing empirical measures in finite-dimensional RKHSs, deriving bounds on coreset sizes, and analyzing the impact of data and kernel properties on compression quality.

Contribution

It introduces new lower bounds on coreset sizes based on data density, kernel properties, and covariance eigenvalues, and extends these results to kernel ridge regression and algorithmic guarantees.

Findings

Lower bounds on coreset sizes depend on data density and kernel conditions.

Eigenvalue bounds of covariance operators inform compression limits.

Standard algorithms like conditional gradient have quantifiable compression guarantees.

Abstract

We study approaches for compressing the empirical measure in the context of finite dimensional reproducing kernel Hilbert spaces (RKHSs). In this context, the empirical measure is contained within a natural convex set and can be approximated using convex optimization methods. Such an approximation gives rise to a coreset of data points. A key quantity that controls how large such a coreset has to be is the size of the largest ball around the empirical measure that is contained within the empirical convex set. The bulk of our work is concerned with deriving high probability lower bounds on the size of such a ball under various conditions and in various settings: we show how conditions on the density of the data and the kernel function can be used to infer such lower bounds; we further develop an approach that uses a lower bound on the smallest eigenvalue of a covariance operator to provide lower bounds on the size of such a ball; we extend the approach to approximate covariance operators and we show how it can be used in the context of kernel ridge regression. We also derive compression guarantees when standard algorithms like the conditional gradient method are used and we discuss variations of such algorithms to improve the runtime of these standard algorithms. We conclude with a construction of an infinite dimensional RKHS for which the compression is poor, highlighting some of the difficulties one faces when trying to move to infinite dimensional RKHSs.