Private Model Personalization Revisited

TL;DR

This paper develops a differentially private federated learning algorithm for model personalization that efficiently recovers shared embeddings and low-dimensional user representations, improving privacy guarantees and extending applicability to noisy labels and broader user distributions.

Contribution

It introduces a privacy-preserving federated learning method for shared embeddings, extending prior work by handling noisy labels and broader distributions, with improved privacy error bounds.

Findings

The proposed algorithm satisfies differential privacy in federated settings.

It achieves improved privacy error bounds compared to previous work.

In binary classification, it provides dimension-independent risk guarantees using Johnson-Lindenstrauss transform.

Abstract

We study model personalization under user-level differential privacy (DP) in the shared representation framework. In this problem, there are $n$ users whose data is statistically heterogeneous, and their optimal parameters share an unknown embedding $U^* \in\mathbb{R}^{d\times k}$ that maps the user parameters in $\mathbb{R}^d$ to low-dimensional representations in $\mathbb{R}^k$, where $k\ll d$. Our goal is to privately recover the shared embedding and the local low-dimensional representations with small excess risk in the federated setting. We propose a private, efficient federated learning algorithm to learn the shared embedding based on the FedRep algorithm in [CHM+21]. Unlike [CHM+21], our algorithm satisfies differential privacy, and our results hold for the case of noisy labels. In contrast to prior work on private model personalization [JRS+21], our utility guarantees hold under a larger class of users' distributions (sub-Gaussian instead of Gaussian distributions). Additionally, in natural parameter regimes, we improve the privacy error term in [JRS+21] by a factor of $\widetilde{O}(dk)$. Next, we consider the binary classification setting. We present an information-theoretic construction to privately learn the shared embedding and derive a margin-based accuracy guarantee that is independent of $d$. Our method utilizes the Johnson-Lindenstrauss transform to reduce the effective dimensions of the shared embedding and the users' data. This result shows that dimension-independent risk bounds are possible in this setting under a margin loss.