COKE: Communication-Censored Decentralized Kernel Learning

TL;DR

This paper introduces COKE, a communication-efficient decentralized kernel learning algorithm that leverages random features and ADMM to enable fast convergence and reduced communication among agents in distributed learning tasks.

Contribution

The paper proposes a novel communication-censored decentralized kernel learning method using random feature approximation and ADMM, with theoretical guarantees and practical efficiency.

Findings

COKE significantly reduces communication load compared to baseline methods.

Theoretical analysis confirms linear convergence and good generalization performance.

Experimental results demonstrate effective learning and communication savings on synthetic and real datasets.

Abstract

This paper studies the decentralized optimization and learning problem where multiple interconnected agents aim to learn an optimal decision function defined over a reproducing kernel Hilbert space by jointly minimizing a global objective function, with access to their own locally observed dataset. As a non-parametric approach, kernel learning faces a major challenge in distributed implementation: the decision variables of local objective functions are data-dependent and thus cannot be optimized under the decentralized consensus framework without any raw data exchange among agents. To circumvent this major challenge, we leverage the random feature (RF) approximation approach to enable consensus on the function modeled in the RF space by data-independent parameters across different agents. We then design an iterative algorithm, termed DKLA, for fast-convergent implementation via ADMM. Based on DKLA, we further develop a communication-censored kernel learning (COKE) algorithm that reduces the communication load of DKLA by preventing an agent from transmitting at every iteration unless its local updates are deemed informative. Theoretical results in terms of linear convergence guarantee and generalization performance analysis of DKLA and COKE are provided. Comprehensive tests on both synthetic and real datasets are conducted to verify the communication efficiency and learning effectiveness of COKE.