On the Saturation Effect of Kernel Ridge Regression

TL;DR

This paper proves a long-standing conjecture that kernel ridge regression cannot surpass a certain lower bound when the true function's smoothness exceeds a specific level, highlighting a fundamental limitation of KRR.

Contribution

It provides a rigorous proof of the saturation effect conjecture for kernel ridge regression, establishing a theoretical lower bound related to function smoothness.

Findings

Confirmed the saturation effect conjecture for KRR

Established a theoretical lower bound for KRR performance

Clarified the limitations of KRR in high-smoothness regimes

Abstract

The saturation effect refers to the phenomenon that the kernel ridge regression (KRR) fails to achieve the information theoretical lower bound when the smoothness of the underground truth function exceeds certain level. The saturation effect has been widely observed in practices and a saturation lower bound of KRR has been conjectured for decades. In this paper, we provide a proof of this long-standing conjecture.