Demystifying Spectral Bias on Real-World Data

TL;DR

This paper investigates how spectral bias affects the learnability of functions in kernel methods on real-world data, using eigenvalues and eigenfunctions to understand the influence of data measures and symmetries.

Contribution

It introduces a framework to analyze spectral bias on complex datasets by relating idealized eigenvalues to real-world data, leveraging kernel symmetries.

Findings

Eigenvalues on idealized data reveal spectral bias in real-world datasets.

Spectral bias influences the learnability of target functions.

Symmetries in kernels help understand spectral bias effects.

Abstract

Kernel ridge regression (KRR) and Gaussian processes (GPs) are fundamental tools in statistics and machine learning, with recent applications to highly over-parameterized deep neural networks. The ability of these tools to learn a target function is directly related to the eigenvalues of their kernel sampled on the input data distribution. Targets that have support on higher eigenvalues are more learnable. However, solving such eigenvalue problems on real-world data remains a challenge. Here, we consider cross-dataset learnability and show that one may use eigenvalues and eigenfunctions associated with highly idealized data measures to reveal spectral bias on complex datasets and bound learnability on real-world data. This allows us to leverage various symmetries that realistic kernels manifest to unravel their spectral bias.