Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

TL;DR

This paper introduces an adaptive kernel learning approach for ridgeless regression, combining theoretical analysis and experiments to explain its generalization and approximation capabilities.

Contribution

It proposes LAB RBF kernels with kernel learning, providing new theoretical insights into their function space and generalization properties without explicit regularization.

Findings

Functions learned belong to an integral space of RKHSs

Optimization is equivalent to an $\, ext{l}_0$-regularized problem

Experimental results validate theoretical analysis

Abstract

Ridgeless regression has garnered attention among researchers, particularly in light of the ``Benign Overfitting'' phenomenon, where models interpolating noisy samples demonstrate robust generalization. However, kernel ridgeless regression does not always perform well due to the lack of flexibility. This paper enhances kernel ridgeless regression with Locally-Adaptive-Bandwidths (LAB) RBF kernels, incorporating kernel learning techniques to improve performance in both experiments and theory. For the first time, we demonstrate that functions learned from LAB RBF kernels belong to an integral space of Reproducible Kernel Hilbert Spaces (RKHSs). Despite the absence of explicit regularization in the proposed model, its optimization is equivalent to solving an $\ell_0$-regularized problem in the integral space of RKHSs, elucidating the origin of its generalization ability. Taking an approximation analysis viewpoint, we introduce an $l_q$-norm analysis technique (with $0<q<1$) to derive the learning rate for the proposed model under mild conditions. This result deepens our theoretical understanding, explaining that our algorithm's robust approximation ability arises from the large capacity of the integral space of RKHSs, while its generalization ability is ensured by sparsity, controlled by the number of support vectors. Experimental results on both synthetic and real datasets validate our theoretical conclusions.