DMM: Distributed Matrix Mechanism for Differentially-Private Federated Learning Based on Constant-Overhead Linear Secret Resharing

TL;DR

This paper introduces a novel cryptographic protocol for federated learning that enhances privacy and utility by combining distributed differential privacy with the matrix mechanism, ensuring secure value transfer with minimal communication overhead.

Contribution

The paper presents a new distributed matrix mechanism and a cryptographic protocol that supports dynamic user participation with constant communication overhead, improving privacy and utility in federated learning.

Findings

Significant utility improvements over previous distributed DP mechanisms.

Secure transfer protocol with constant communication overhead.

Supports dynamic user participation including dropouts.

Abstract

Federated Learning (FL) solutions with central Differential Privacy (DP) have seen large improvements in their utility in recent years arising from the matrix mechanism, while FL solutions with distributed (more private) DP have lagged behind. In this work, we introduce the distributed matrix mechanism to achieve the best-of-both-worlds; better privacy of distributed DP and better utility from the matrix mechanism. We accomplish this using a novel cryptographic protocol that securely transfers sensitive values across client committees of different training iterations with constant communication overhead. This protocol accommodates the dynamic participation of users required by FL, including those that may drop out from the computation. We provide experiments which show that our mechanism indeed significantly improves the utility of FL models compared to previous distributed DP mechanisms, with little added overhead.