Decentralized Kernel Ridge Regression Based on Data-Dependent Random Feature

TL;DR

This paper introduces a decentralized kernel ridge regression method that uses data-dependent random features, allowing for adaptive local data handling while maintaining communication efficiency, leading to significant accuracy improvements.

Contribution

It proposes a novel decentralized KRR algorithm with data-dependent random features, enabling adaptive consensus on decision functions across nodes.

Findings

Achieved 25.5% average regression accuracy improvement

Maintained same communication costs as existing methods

Validated effectiveness on six real-world datasets

Abstract

Random feature (RF) has been widely used for node consistency in decentralized kernel ridge regression (KRR). Currently, the consistency is guaranteed by imposing constraints on coefficients of features, necessitating that the random features on different nodes are identical. However, in many applications, data on different nodes varies significantly on the number or distribution, which calls for adaptive and data-dependent methods that generate different RFs. To tackle the essential difficulty, we propose a new decentralized KRR algorithm that pursues consensus on decision functions, which allows great flexibility and well adapts data on nodes. The convergence is rigorously given and the effectiveness is numerically verified: by capturing the characteristics of the data on each node, while maintaining the same communication costs as other methods, we achieved an average regression accuracy improvement of 25.5\% across six real-world data sets.