LoRA and Privacy: When Random Projections Help (and When They Don't)

TL;DR

This paper investigates the differential privacy properties of a randomized projection mechanism, demonstrating its strengths for vector queries, limitations for matrix queries, and how low-rank projections can enhance privacy in machine learning fine-tuning.

Contribution

The paper introduces the Wishart projection mechanism, analyzes its privacy guarantees for vector and matrix queries, and connects LoRA updates to this mechanism, revealing privacy trade-offs in low-rank fine-tuning.

Findings

Wishart projections provide DP guarantees for vector queries without noise.

The mechanism is not DP for matrix queries without added noise, vulnerable to membership inference.

Low-rank projections can amplify privacy, outperforming additive noise in certain regimes.

Abstract

We introduce the (Wishart) projection mechanism, a randomized map of the form $S \mapsto M f(S)$ with $M \sim W_d(1/r I_d, r)$ and study its differential privacy properties. For vector-valued queries $f$, we prove non-asymptotic DP guarantees without any additive noise, showing that Wishart randomness alone can suffice. For matrix-valued queries, however, we establish a sharp negative result: in the noise-free setting, the mechanism is not DP, and we demonstrate its vulnerability by implementing a near perfect membership inference attack (AUC $> 0.99$). We then analyze a noisy variant and prove privacy amplification due to randomness and low rank projection, in both large- and small-rank regimes, yielding stronger privacy guarantees than additive noise alone. Finally, we show that LoRA-style updates are an instance of the matrix-valued mechanism, implying that LoRA is not inherently private despite its built-in randomness, but that low-rank fine-tuning can be more private than full fine-tuning at the same noise level. Preliminary experiments suggest that tighter accounting enables lower noise and improved accuracy in practice.