Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimization

TL;DR

This paper enhances prefill attacks on large language models by ensembling variants and introduces sockpuppetting, a hybrid attack optimizing adversarial suffixes, revealing vulnerabilities in open-weight models.

Contribution

It demonstrates improved attack success rates through ensembling and introduces sockpuppetting, a novel hybrid attack method targeting output-prefix injection vulnerabilities.

Findings

Ensembling small prefill variants boosts attack success rates significantly.

Sockpuppetting increases prompt-agnostic attack success by up to 64%.

Code for the attacks is publicly available at the provided GitHub URL.

Abstract

Prefill attacks are an effective and low-cost jailbreaking method, as they directly insert an acceptance sequence (e.g., "Sure, here is how to...") at the start of an LLM's output and lead the model to continue the response. We make two contributions to this prior work. First, we show that an unsophisticated adversary can improve the well-known prefill attacks by ensembling a small number of prefill variants. Running three easy-to-generate prefills yields a combined attack success rate (ASR) of 22%, 90%, and 99% on Gemma-7B, Llama-3.1-8B, and Qwen3-8B respectively, an up to 38% improvement over the standard "Sure, here's..." prefill and up to 82% over our reproduction of GCG (Zou et al., 2023). Second, we introduce "sockpuppetting", a hybrid attack that optimizes an adversarial suffix placed inside the "assistant" message block of the chat template, rather than within the user prompt. The rolling variant of this attack, RollingSockpuppetGCG, increases prompt-agnostic ASR by up to 64% over our universal GCG baseline on Llama-3.1-8B. Both findings highlight the need for defences against output-prefix injection in open-weight models. Code: https://gitlab.com/asendotsinski/sockpuppetting