(Amplified) Banded Matrix Factorization: A unified approach to private training

TL;DR

This paper introduces a unified matrix factorization approach with banded matrices for differential privacy, improving privacy-utility tradeoffs in federated and centralized machine learning, and demonstrating practical deployment benefits.

Contribution

It presents a novel banded matrix mechanism that subsumes prior algorithms, offering better privacy-utility tradeoffs and practical federated learning deployment.

Findings

Banded matrix mechanisms match or outperform DP-SGD in privacy-utility tradeoffs.

Enables multiple participations in federated learning with relaxed device schemas.

Proven privacy amplification for banded matrices similar to DP-SGD.

Abstract

Matrix factorization (MF) mechanisms for differential privacy (DP) have substantially improved the state-of-the-art in privacy-utility-computation tradeoffs for ML applications in a variety of scenarios, but in both the centralized and federated settings there remain instances where either MF cannot be easily applied, or other algorithms provide better tradeoffs (typically, as $\epsilon$ becomes small). In this work, we show how MF can subsume prior state-of-the-art algorithms in both federated and centralized training settings, across all privacy budgets. The key technique throughout is the construction of MF mechanisms with banded matrices (lower-triangular matrices with at most $\hat{b}$ nonzero bands including the main diagonal). For cross-device federated learning (FL), this enables multiple-participations with a relaxed device participation schema compatible with practical FL infrastructure (as demonstrated by a production deployment). In the centralized setting, we prove that banded matrices enjoy the same privacy amplification results as the ubiquitous DP-SGD algorithm, but can provide strictly better performance in most scenarios -- this lets us always at least match DP-SGD, and often outperform it.