Secure Aggregation Meets Sparsification in Decentralized Learning

TL;DR

This paper presents CESAR, a secure aggregation protocol compatible with sparsification in decentralized learning, enhancing privacy and communication efficiency while maintaining high model accuracy.

Contribution

CESAR is a novel secure aggregation protocol designed for sparsified decentralized learning, providing provable security and analytical insights into its interaction with sparsification.

Findings

CESAR maintains within 0.5% accuracy of D-PSGD with 11% data overhead.

CESAR outperforms TopK accuracy by up to 0.3% on IID data.

The protocol is secure against honest-but-curious adversaries and adaptable to collusion scenarios.

Abstract

Decentralized learning (DL) faces increased vulnerability to privacy breaches due to sophisticated attacks on machine learning (ML) models. Secure aggregation is a computationally efficient cryptographic technique that enables multiple parties to compute an aggregate of their private data while keeping their individual inputs concealed from each other and from any central aggregator. To enhance communication efficiency in DL, sparsification techniques are used, selectively sharing only the most crucial parameters or gradients in a model, thereby maintaining efficiency without notably compromising accuracy. However, applying secure aggregation to sparsified models in DL is challenging due to the transmission of disjoint parameter sets by distinct nodes, which can prevent masks from canceling out effectively. This paper introduces CESAR, a novel secure aggregation protocol for DL designed to be compatible with existing sparsification mechanisms. CESAR provably defends against honest-but-curious adversaries and can be formally adapted to counteract collusion between them. We provide a foundational understanding of the interaction between the sparsification carried out by the nodes and the proportion of the parameters shared under CESAR in both colluding and non-colluding environments, offering analytical insight into the working and applicability of the protocol. Experiments on a network with 48 nodes in a 3-regular topology show that with random subsampling, CESAR is always within 0.5% accuracy of decentralized parallel stochastic gradient descent (D-PSGD), while adding only 11% of data overhead. Moreover, it surpasses the accuracy on TopK by up to 0.3% on independent and identically distributed (IID) data.