Adaptivity for Regularized Kernel Methods by Lepskii's Principle

TL;DR

This paper demonstrates that the Lepskii Principle can be effectively used for adaptive regularization parameter selection in kernel regression, achieving near-optimal minimax rates without prior structural knowledge.

Contribution

It introduces a modified Lepskii-based parameter choice method that is proven to be minimax optimal up to a logarithmic factor in RKHS regression.

Findings

Lepskii Principle achieves near-minimax optimal adaptivity.

Balancing in L^2 norm ensures optimality in stronger norms.

The method outperforms classical approaches like Hold-Out in adaptivity.

Abstract

We address the problem of {\it adaptivity} in the framework of reproducing kernel Hilbert space (RKHS) regression. More precisely, we analyze estimators arising from a linear regularization scheme $g_\lam$. In practical applications, an important task is to choose the regularization parameter $\lam$ appropriately, i.e. based only on the given data and independently on unknown structural assumptions on the regression function. An attractive approach avoiding data-splitting is the {\it Lepskii Principle} (LP), also known as the {\it Balancing Principle} is this setting. We show that a modified parameter choice based on (LP) is minimax optimal adaptive, up to $\log\log(n)$. A convenient result is the fact that balancing in $L^2(\nu)-$ norm, which is easiest, automatically gives optimal balancing in all stronger norms, interpolating between $L^2(\nu)$ and the RKHS. An analogous result is open for other classical approaches to data dependent choices of the regularization parameter, e.g. for Hold-Out.