Federated Transfer Learning with Differential Privacy

TL;DR

This paper introduces federated transfer learning with differential privacy, addressing data heterogeneity and privacy, and analyzes statistical problems to understand privacy costs and knowledge transfer benefits.

Contribution

It formulates federated differential privacy, studies its impact on statistical estimation, and compares it to local and central privacy models, highlighting fundamental costs and advantages.

Findings

Federated differential privacy is an intermediate privacy model.

Analyzed minimax rates for various statistical problems under privacy constraints.

Quantified the privacy-utility trade-offs and benefits of knowledge transfer.

Abstract

Federated learning has emerged as a powerful framework for analysing distributed data, yet two challenges remain pivotal: heterogeneity across sites and privacy of local data. In this paper, we address both challenges within a federated transfer learning framework, aiming to enhance learning on a target data set by leveraging information from multiple heterogeneous source data sets while adhering to privacy constraints. We rigorously formulate the notion of federated differential privacy, which offers privacy guarantees for each data set without assuming a trusted central server. Under this privacy model, we study four statistical problems: univariate mean estimation, low-dimensional linear regression, high-dimensional linear regression, and M-estimation. By investigating the minimax rates and quantifying the cost of privacy, we show that federated differential privacy is an intermediate privacy model between the well-established local and central models of differential privacy. Our analyses account for data heterogeneity and privacy, highlighting the fundamental costs associated with each factor and the benefits of knowledge transfer in federated learning.