Target alignment in truncated kernel ridge regression

TL;DR

This paper investigates how target-kernel alignment influences the performance of truncated kernel ridge regression, revealing that spectral truncation can enhance learning rates and induce complex generalization behaviors.

Contribution

It introduces the concept of over-alignment in polynomial settings and demonstrates how spectral truncation in TKRR can outperform full KRR in certain regimes.

Findings

Over-aligned regime allows TKRR to achieve parametric rates.

Spectral truncation can induce multiple descent and non-monotonic generalization curves.

Alignment spectrum shape strongly influences kernel method performance.

Abstract

Kernel ridge regression (KRR) has recently attracted renewed interest due to its potential for explaining the transient effects, such as double descent, that emerge during neural network training. In this work, we study how the alignment between the target function and the kernel affects the performance of the KRR. We focus on the truncated KRR (TKRR) which utilizes an additional parameter that controls the spectral truncation of the kernel matrix. We show that for polynomial alignment, there is an \emph{over-aligned} regime, in which TKRR can achieve a faster rate than what is achievable by full KRR. The rate of TKRR can improve all the way to the parametric rate, while that of full KRR is capped at a sub-optimal value. This shows that target alignemnt can be better leveraged by utilizing spectral truncation in kernel methods. We also consider the bandlimited alignment setting and show that the regularization surface of TKRR can exhibit transient effects including multiple descent and non-monotonic behavior. Our results show that there is a strong and quantifable relation between the shape of the \emph{alignment spectrum} and the generalization performance of kernel methods, both in terms of rates and in finite samples.