Scaling up the Banded Matrix Factorization Mechanism for Differentially Private ML

TL;DR

This paper introduces scalable techniques for the DP-BandMF mechanism, enabling it to handle large-scale models and training iterations in differentially private machine learning without significant utility loss.

Contribution

We develop methods to significantly scale up DP-BandMF, overcoming previous limitations in handling large models and many training iterations.

Findings

Enabled DP-BandMF to handle models with over 10^7 parameters.

Supported training with over 10^4 iterations.

Maintained utility with negligible degradation.

Abstract

Correlated noise mechanisms such as DP Matrix Factorization (DP-MF) have proven to be effective alternatives to DP-SGD in large-epsilon few-epoch training regimes. Significant work has been done to find the best correlated noise strategies, and the current state-of-the-art approach is DP-BandMF, which optimally balances the benefits of privacy amplification and noise correlation. Despite it's utility advantages, severe scalability limitations prevent this mechanism from handling large-scale training scenarios where the number of training iterations may exceed $10^4$ and the number of model parameters may exceed $10^7$. In this work, we present techniques to scale up DP-BandMF along these two dimensions, significantly extending it's reach and enabling it to handle settings with virtually any number of model parameters and training iterations, with negligible utility degradation.