Beyond Square Roots: Explicit Memory-Efficient Factorization for Multi-Epoch Private Learning

TL;DR

This paper introduces $\gamma$-BIFR, a new memory-efficient factorization method that improves private learning utility and guarantees across different bandwidth regimes.

Contribution

It unifies existing banded inverse factorizations into a single framework, enhancing performance in low-memory, multi-epoch private training.

Findings

$\gamma$-BIFR improves RMSE and private training performance in low-memory regimes.

Provides tighter theoretical guarantees for multi-participation error.

Outperforms existing factorizations at low bandwidths.

Abstract

Correlated-noise mechanisms are among the most promising approaches for improving the utility of differentially private model training, but rigorous guarantees require explicit, analyzable factorizations, and practical deployment requires memory efficiency. Recent works have developed banded inverse factorizations, which address both requirements by exploiting a banded structure in the correlation matrix. The bandwidth controls the size of the noise buffer used to correlate noise across iterations, and thus governs the tradeoff between utility and memory cost. Existing factorizations highlight this tradeoff: DP-$\lambda$CGD achieves high memory efficiency by using only a one-step noise buffer, but this limits its utility gains, while the banded inverse square root (BISR) factorization exploits larger correlation windows and is asymptotically optimal for large bandwidths but performs poorly at low bandwidths. We propose $\gamma$-BIFR, a unified generalization of both factorizations. In the low-memory, low-bandwidth regime, $\gamma$-BIFR significantly improves RMSE, amplified RMSE, and private training performance, while yielding tighter theoretical guarantees for multi-participation error in multi-epoch training.