P4: Towards private, personalized, and Peer-to-Peer learning

TL;DR

This paper introduces P4, a decentralized learning method that achieves personalized, private models through client clustering and knowledge distillation, outperforming existing private P2P approaches on benchmark datasets.

Contribution

P4 is the first approach to combine client clustering and differential privacy in a decentralized, peer-to-peer learning framework with minimal overhead.

Findings

P4 outperforms state-of-the-art private P2P methods by up to 40% in accuracy.

The method maintains differential privacy guarantees during and after training.

P4 is practical for resource-constrained devices with minimal training overhead.

Abstract

Personalized learning is a proposed approach to address the problem of data heterogeneity in collaborative machine learning. In a decentralized setting, the two main challenges of personalization are client clustering and data privacy. In this paper, we address these challenges by developing P4 (Personalized Private Peer-to-Peer) a method that ensures that each client receives a personalized model while maintaining differential privacy guarantee of each client's local dataset during and after the training. Our approach includes the design of a lightweight algorithm to identify similar clients and group them in a private, peer-to-peer (P2P) manner. Once grouped, we develop differentially-private knowledge distillation for clients to co-train with minimal impact on accuracy. We evaluate our proposed method on three benchmark datasets (FEMNIST or Federated EMNIST, CIFAR-10 and CIFAR-100) and two different neural network architectures (Linear and CNN-based networks) across a range of privacy parameters. The results demonstrate the potential of P4, as it outperforms the state-of-the-art of differential private P2P by up to 40 percent in terms of accuracy. We also show the practicality of P4 by implementing it on resource constrained devices, and validating that it has minimal overhead, e.g., about 7 seconds to run collaborative training between two clients.