Voting-based Approaches For Differentially Private Federated Learning

TL;DR

This paper introduces voting-based methods for Differentially Private Federated Learning that reduce communication costs and improve privacy guarantees by replacing gradient averaging with label voting, demonstrating superior privacy-utility trade-offs.

Contribution

The paper proposes two novel DPFL schemes using voting among local model labels, avoiding dimension dependence and enhancing privacy amplification.

Findings

Significant improvement in privacy-utility trade-off over existing methods

Reduction in communication cost compared to gradient averaging approaches

Exponential privacy amplification achieved through secure multi-party computation

Abstract

Differentially Private Federated Learning (DPFL) is an emerging field with many applications. Gradient averaging based DPFL methods require costly communication rounds and hardly work with large-capacity models, due to the explicit dimension dependence in its added noise. In this work, inspired by knowledge transfer non-federated privacy learning from Papernot et al.(2017; 2018), we design two new DPFL schemes, by voting among the data labels returned from each local model, instead of averaging the gradients, which avoids the dimension dependence and significantly reduces the communication cost. Theoretically, by applying secure multi-party computation, we could exponentially amplify the (data-dependent) privacy guarantees when the margin of the voting scores are large. Extensive experiments show that our approaches significantly improve the privacy-utility trade-off over the state-of-the-arts in DPFL.