A general technique for approximating high-dimensional empirical kernel matrices

TL;DR

This paper introduces a unified approach using decoupling and non-commutative inequalities to bound and approximate high-dimensional empirical kernel matrices, improving existing results and providing new insights for Gaussian data.

Contribution

The authors develop a simple, general technique for bounding and approximating kernel matrices, offering tighter bounds and novel results for anisotropic Gaussian data.

Findings

Provided new, tighter bounds for kernel matrix operator norms.

Achieved simplified proofs of existing results using decoupling and inequalities.

Derived novel approximation results and bias bounds for Gaussian kernel regression.

Abstract

We present simple, user-friendly bounds for the expected operator norm of a random kernel matrix under general conditions on the kernel function $k(\cdot,\cdot)$. Our approach uses decoupling results for U-statistics and the non-commutative Khintchine inequality to obtain upper and lower bounds depending only on scalar statistics of the kernel function and a ``correlation kernel'' matrix corresponding to $k(\cdot,\cdot)$. We then apply our method to provide new, tighter approximations for inner-product kernel matrices on general high-dimensional data, where the sample size and data dimension are polynomially related. Our method obtains simplified proofs of existing results that rely on the moment method and combinatorial arguments while also providing novel approximation results for the case of anisotropic Gaussian data. Finally, using similar techniques to our approximation result, we show a tighter lower bound on the bias of kernel regression with anisotropic Gaussian data.