On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference

TL;DR

This paper demonstrates that the shuffling defense in secure Transformer inference is vulnerable, enabling an attack to recover model weights with minimal queries and high accuracy, challenging previous security claims.

Contribution

The authors introduce a novel attack that effectively breaks the shuffling defense, revealing its weaknesses and raising concerns about the security of secure inference methods.

Findings

The attack achieves mean squared errors between $10^{-9}$ and $10^{-6}$ on shuffled activations.

With about $1 worth of queries, the attack recovers model weights with L1-norm differences of $10^{-4}$ to $10^{-2}$.

The shuffling defense is less robust than previously claimed, exposing vulnerabilities in secure inference.

Abstract

For Transformer models, cryptographically secure inference ensures that the client learns only the final output, while the server learns nothing about the client's input. However, securely computing nonlinear layers remains a major efficiency bottleneck due to the substantial communication rounds and data transmission required. To address this issue, prior works reveal intermediate activations to the client, allowing nonlinear operations to be computed in plaintext. Although this approach significantly improves efficiency, exposing activations enables adversaries to extract model weights. To mitigate this risk, existing works employ a shuffling defense that reveals only randomly permuted activations to the client. In this work, we show that the shuffling defense is not as robust as previously claimed. We propose an attack that aligns differently shuffled activations to a common permutation and subsequently exploits them to extract model weights. Experiments on Pythia-70m and GPT-2 demonstrate that the proposed attack can align shuffled activations with mean squared errors ranging from $10^{-9}$ to $10^{-6}$. With a query cost of approximately \$1, the adversary can recover model weights with L1-norm differences ranging from $10^{-4}$ to $10^{-2}$ compared to the oracle weights.