Harnessing Sparsification in Federated Learning: A Secure, Efficient, and Differentially Private Realization

TL;DR

Clover is a system framework that enhances federated learning by combining gradient sparsification, secure aggregation, and differential privacy, significantly reducing communication costs and improving privacy protections while maintaining model utility.

Contribution

It introduces a novel secure aggregation mechanism for sparse gradients in federated learning, outperforming existing methods in efficiency and privacy guarantees.

Findings

Clover reduces server communication and runtime by orders of magnitude.

It achieves differential privacy with utility comparable to centralized DP.

The system maintains high model accuracy with enhanced security and privacy.

Abstract

Federated learning (FL) enables multiple clients to jointly train a model by sharing only gradient updates for aggregation instead of raw data. Due to the transmission of very high-dimensional gradient updates from many clients, FL is known to suffer from a communication bottleneck. Meanwhile, the gradients shared by clients as well as the trained model may also be exploited for inferring private local datasets, making privacy still a critical concern in FL. We present Clover, a novel system framework for communication-efficient, secure, and differentially private FL. To tackle the communication bottleneck in FL, Clover follows a standard and commonly used approach-top-k gradient sparsification, where each client sparsifies its gradient update such that only k largest gradients (measured by magnitude) are preserved for aggregation. Clover provides a tailored mechanism built out of a trending distributed trust setting involving three servers, which allows to efficiently aggregate multiple sparse vectors (top-k sparsified gradient updates) into a dense vector while hiding the values and indices of non-zero elements in each sparse vector. This mechanism outperforms a baseline built on the general distributed ORAM technique by several orders of magnitude in server-side communication and runtime, with also smaller client communication cost. We further integrate this mechanism with a lightweight distributed noise generation mechanism to offer differential privacy (DP) guarantees on the trained model. To harden Clover with security against a malicious server, we devise a series of lightweight mechanisms for integrity checks on the server-side computation. Extensive experiments show that Clover can achieve utility comparable to vanilla FL with central DP, with promising performance.