Stealing User Prompts from Mixture of Experts

TL;DR

This paper demonstrates a novel attack exploiting architectural flaws in Mixture-of-Experts models, enabling full prompt disclosure through carefully arranged queries, revealing a new class of vulnerabilities in language models.

Contribution

It introduces the first attack exploiting architectural flaws in MoE models to extract user prompts, highlighting a new security vulnerability.

Findings

Successfully extracted entire prompts with $O({VM}^2)$ queries

Achieved prompt extraction with an average of 100 queries per token

Demonstrated attack effectiveness on a two-layer Mixtral model

Abstract

Mixture-of-Experts (MoE) models improve the efficiency and scalability of dense language models by routing each token to a small number of experts in each layer. In this paper, we show how an adversary that can arrange for their queries to appear in the same batch of examples as a victim's queries can exploit Expert-Choice-Routing to fully disclose a victim's prompt. We successfully demonstrate the effectiveness of this attack on a two-layer Mixtral model, exploiting the tie-handling behavior of the torch.topk CUDA implementation. Our results show that we can extract the entire prompt using $O({VM}^2)$ queries (with vocabulary size $V$ and prompt length $M$) or 100 queries on average per token in the setting we consider. This is the first attack to exploit architectural flaws for the purpose of extracting user prompts, introducing a new class of LLM vulnerabilities.