Lap2: Revisiting Laplace DP-SGD for High Dimensions via Majorization Theory

TL;DR

This paper introduces Lap2, a novel method for L2 clipping in Laplace DP-SGD that overcomes high-dimensional limitations, enabling privacy-preserving training of large models with improved utility.

Contribution

Lap2 leverages majorization theory and coordinate-wise bounds to enable effective L2 clipping in high-dimensional Laplace DP-SGD, improving privacy utility trade-offs.

Findings

Achieves comparable or better accuracy than Gaussian DP-SGD under strong privacy constraints.

Enables privacy-preserving fine-tuning of large models like RoBERTa-base with high utility.

Scales gracefully with model dimension, handling thousands of moments.

Abstract

Differentially Private Stochastic Gradient Descent (DP-SGD) is a cornerstone technique for ensuring privacy in deep learning, widely used in both training from scratch and fine-tuning large-scale language models. While DP-SGD predominantly relies on the Gaussian mechanism, the Laplace mechanism remains underutilized due to its reliance on L1 norm clipping. This constraint severely limits its practicality in high-dimensional models because the L1 norm of an n-dimensional gradient can be up to sqrt(n) times larger than its L2 norm. As a result, the required noise scale grows significantly with model size, leading to poor utility or untrainable models. In this work, we introduce Lap2, a new solution that enables L2 clipping for Laplace DP-SGD while preserving strong privacy guarantees. We overcome the dimensionality-driven clipping barrier by computing coordinate-wise moment bounds and applying majorization theory to construct a tight, data-independent upper bound over the full model. By exploiting the Schur-convexity of the moment accountant function, we aggregate these bounds using a carefully designed majorization set that respects the L2 clipping constraint. This yields a multivariate privacy accountant that scales gracefully with model dimension and enables the use of thousands of moments. Empirical evaluations demonstrate that our approach significantly improves the performance of Laplace DP-SGD, achieving results comparable to or better than Gaussian DP-SGD under strong privacy constraints. For instance, fine-tuning RoBERTa-base (125M parameters) on SST-2 achieves 87.88% accuracy at epsilon=0.54, outperforming Gaussian (87.16%) and standard Laplace (48.97%) under the same budget.