Kernel-based L_2-Boosting with Structure Constraints

TL;DR

This paper introduces KReBooT, a kernel-based boosting algorithm that controls estimator structure, achieves near-optimal convergence, and demonstrates strong theoretical and numerical performance in regression tasks.

Contribution

It proposes a novel kernel-based boosting method with structure control and near overfitting resistance, backed by theoretical convergence and learning rate guarantees.

Findings

Achieves almost optimal convergence rate for nonlinear approximation

Provides fast learning rates using integral operator and concentration inequalities

Demonstrates promising numerical performance with good generalization

Abstract

Developing efficient kernel methods for regression is very popular in the past decade. In this paper, utilizing boosting on kernel-based weaker learners, we propose a novel kernel-based learning algorithm called kernel-based re-scaled boosting with truncation, dubbed as KReBooT. The proposed KReBooT benefits in controlling the structure of estimators and producing sparse estimate, and is near overfitting resistant. We conduct both theoretical analysis and numerical simulations to illustrate the power of KReBooT. Theoretically, we prove that KReBooT can achieve the almost optimal numerical convergence rate for nonlinear approximation. Furthermore, using the recently developed integral operator approach and a variant of Talagrand's concentration inequality, we provide fast learning rates for KReBooT, which is a new record of boosting-type algorithms. Numerically, we carry out a series of simulations to show the promising performance of KReBooT in terms of its good generalization, near over-fitting resistance and structure constraints.