Enhancing the Privacy of Federated Learning with Sketching

TL;DR

This paper proposes using sketching algorithms to enhance privacy in federated learning, achieving strong privacy guarantees without compromising performance or accuracy, and explores the fundamental link between privacy and communication.

Contribution

It introduces sketching algorithms as a novel approach to improve privacy and efficiency in federated learning, addressing key trade-offs in existing methods.

Findings

Sketching algorithms can provide both privacy and performance benefits.

Strong privacy guarantees are achievable without sacrificing accuracy.

The work reveals a fundamental connection between privacy and communication in distributed learning.

Abstract

In response to growing concerns about user privacy, federated learning has emerged as a promising tool to train statistical models over networks of devices while keeping data localized. Federated learning methods run training tasks directly on user devices and do not share the raw user data with third parties. However, current methods still share model updates, which may contain private information (e.g., one's weight and height), during the training process. Existing efforts that aim to improve the privacy of federated learning make compromises in one or more of the following key areas: performance (particularly communication cost), accuracy, or privacy. To better optimize these trade-offs, we propose that \textit{sketching algorithms} have a unique advantage in that they can provide both privacy and performance benefits while maintaining accuracy. We evaluate the feasibility of sketching-based federated learning with a prototype on three representative learning models. Our initial findings show that it is possible to provide strong privacy guarantees for federated learning without sacrificing performance or accuracy. Our work highlights that there exists a fundamental connection between privacy and communication in distributed settings, and suggests important open problems surrounding the theoretical understanding, methodology, and system design of practical, private federated learning.