SOMP: Scalable Gradient Inversion for Large Language Models via Subspace-Guided Orthogonal Matching Pursuit

TL;DR

SOMP introduces a scalable gradient inversion method that exploits geometric structures in transformer gradients to effectively reconstruct private training text from shared gradients, even at large batch sizes.

Contribution

The paper proposes SOMP, a novel sparse signal recovery framework that leverages head-wise geometric structure and sample sparsity to improve text reconstruction from aggregated gradients in large language models.

Findings

Outperforms prior methods in various LLMs and languages.

Achieves higher fidelity in long sequence reconstructions at large batch sizes.

Reveals persistent privacy risks even under extreme gradient aggregation.

Abstract

Gradient inversion attacks reveal that private training text can be reconstructed from shared gradients, posing a privacy risk to large language models (LLMs). While prior methods perform well in small-batch settings, scaling to larger batch sizes and longer sequences remains challenging due to severe signal mixing, high computational cost, and degraded fidelity. We present SOMP (Subspace-Guided Orthogonal Matching Pursuit), a scalable gradient inversion framework that casts text recovery from aggregated gradients as a sparse signal recovery problem. Our key insight is that aggregated transformer gradients retain exploitable head-wise geometric structure together with sample-level sparsity. SOMP leverages these properties to progressively narrow the search space and disentangle mixed signals without exhaustive search. Experiments across multiple LLM families, model scales, and five languages show that SOMP consistently outperforms prior methods in the aggregated-gradient regime.For long sequences at batch size B=16, SOMP achieves substantially higher reconstruction fidelity than strong baselines, while remaining computationally competitive. Even under extreme aggregation (up to B=128), SOMP still recovers meaningful text, suggesting that privacy leakage can persist in regimes where prior attacks become much less effective.