Locally Differentially Private Distributed Online Learning with Guaranteed Optimality

TL;DR

This paper introduces a novel distributed online learning algorithm that guarantees both local differential privacy and learning accuracy, effectively balancing privacy protection with model performance over infinite time horizons.

Contribution

It presents the first algorithm to ensure rigorous local differential privacy alongside learning accuracy in distributed online learning settings.

Findings

Achieves diminishing expected instantaneous regret.

Ensures finite cumulative privacy budget over infinite horizon.

Validated on logistic regression and CNN image classification tasks.

Abstract

Distributed online learning is gaining increased traction due to its unique ability to process large-scale datasets and streaming data. To address the growing public awareness and concern on privacy protection, plenty of algorithms have been proposed to enable differential privacy in distributed online optimization and learning. However, these algorithms often face the dilemma of trading learning accuracy for privacy. By exploiting the unique characteristics of online learning, this paper proposes an approach that tackles the dilemma and ensures both differential privacy and learning accuracy in distributed online learning. More specifically, while ensuring a diminishing expected instantaneous regret, the approach can simultaneously ensure a finite cumulative privacy budget, even in the infinite time horizon. To cater for the fully distributed setting, we adopt the local differential-privacy framework, which avoids the reliance on a trusted data curator that is required in the classic "centralized" (global) differential-privacy framework. To the best of our knowledge, this is the first algorithm that successfully ensures both rigorous local differential privacy and learning accuracy. The effectiveness of the proposed algorithm is evaluated using machine learning tasks, including logistic regression on the the "mushrooms" datasets and CNN-based image classification on the "MNIST" and "CIFAR-10" datasets.