On Privacy and Personalization in Cross-Silo Federated Learning

TL;DR

This paper explores privacy-preserving techniques in cross-silo federated learning, proposing a silo-specific sample-level differential privacy approach and demonstrating the effectiveness of mean-regularized multi-task learning for personalization under privacy constraints.

Contribution

It introduces a new privacy notion suitable for cross-silo FL and shows that mean-regularized multi-task learning is a strong baseline under privacy constraints.

Findings

Stronger privacy leads silos to federate more, reducing DP noise impact.

MR-MTL improves performance over standard methods under privacy constraints.

Theoretical analysis clarifies the privacy-heterogeneity trade-off.

Abstract

While the application of differential privacy (DP) has been well-studied in cross-device federated learning (FL), there is a lack of work considering DP and its implications for cross-silo FL, a setting characterized by a limited number of clients each containing many data subjects. In cross-silo FL, usual notions of client-level DP are less suitable as real-world privacy regulations typically concern the in-silo data subjects rather than the silos themselves. In this work, we instead consider an alternative notion of silo-specific sample-level DP, where silos set their own privacy targets for their local examples. Under this setting, we reconsider the roles of personalization in federated learning. In particular, we show that mean-regularized multi-task learning (MR-MTL), a simple personalization framework, is a strong baseline for cross-silo FL: under stronger privacy requirements, silos are incentivized to federate more with each other to mitigate DP noise, resulting in consistent improvements relative to standard baseline methods. We provide an empirical study of competing methods as well as a theoretical characterization of MR-MTL for mean estimation, highlighting the interplay between privacy and cross-silo data heterogeneity. Our work serves to establish baselines for private cross-silo FL as well as identify key directions of future work in this area.