Theory of Decentralized Robust Kernel-Based Learning

TL;DR

This paper introduces a decentralized robust kernel learning algorithm in RKHS that unifies robust regression with convergence guarantees, enhancing robustness and efficiency in distributed settings.

Contribution

It presents a novel decentralized robust kernel-based learning framework with rigorous convergence analysis and optimal learning rates, differing from existing divide-and-conquer approaches.

Findings

The algorithm achieves near-optimal convergence rates.

Local estimators effectively approximate the regression function.

Robustness is enhanced through the parameter , improving convergence behavior.

Abstract

We propose a new decentralized robust kernel-based learning algorithm within the framework of reproducing kernel Hilbert spaces (RKHSs) by utilizing a networked system that can be represented as a connected graph. The robust loss function $\huaL_\sigma$ induced by a windowing function $W$ and a robustness scaling parameter $\sigma>0$ can encompass a broad spectrum of robust losses. Consequently, the proposed algorithm effectively provides a unified decentralized learning framework for robust regression, which fundamentally differs from the existing distributed robust kernel-based learning schemes, all of which are divide-and-conquer based. We rigorously establish a learning theory and offer comprehensive convergence analysis for the algorithm. We show each local robust estimator generated from the decentralized algorithm can be utilized to approximate the regression function. Based on kernel-based integral operator techniques, we derive general high confidence convergence bounds for the local approximating sequence in terms of the mean square distance, RKHS norm, and generalization error, respectively. Moreover, we provide rigorous selection rules for local sample size and show that, under properly selected step size and scaling parameter $\sigma$, the decentralized robust algorithm can achieve optimal learning rates (up to logarithmic factors) in both norms. The parameter $\sigma$ is shown to be essential for enhancing robustness and ensuring favorable convergence behavior. The intrinsic connection among decentralization, sample selection, robustness of the algorithm, and its convergence is clearly reflected.