Differentially Private Distributed Online Learning

TL;DR

This paper introduces a differentially private distributed online learning algorithm that enhances privacy and convergence speed in large-scale data scenarios, with practical validation through simulations.

Contribution

It proposes a novel distributed online learning framework incorporating differential privacy, achieving tighter utility bounds and faster convergence than existing methods.

Findings

Differential privacy effectively preserves data privacy in distributed online learning.

Mini-batch techniques improve the high variance issue in private offline learning.

The proposed framework demonstrates accurate theoretical and simulation results.

Abstract

Online learning has been in the spotlight from the machine learning society for a long time. To handle massive data in Big Data era, one single learner could never efficiently finish this heavy task. Hence, in this paper, we propose a novel distributed online learning algorithm to solve the problem. Comparing to typical centralized online learner, the distributed learners optimize their own learning parameters based on local data sources and timely communicate with neighbors. However, communication may lead to a privacy breach. Thus, we use differential privacy to preserve the privacy of learners, and study the influence of guaranteeing differential privacy on the utility of the distributed online learning algorithm. Furthermore, by using the results from Kakade and Tewari (2009), we use the regret bounds of online learning to achieve fast convergence rates for offline learning algorithms in distributed scenarios, which provides tighter utility performance than the existing state-of-the-art results. In simulation, we demonstrate that the differentially private offline learning algorithm has high variance, but we can use mini-batch to improve the performance. Finally, the simulations show that the analytical results of our proposed theorems are right and our private distributed online learning algorithm is a general framework.