Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundaries

TL;DR

This paper uncovers a hidden vulnerability in aligned LLMs where appending eos tokens can cause context segmentation, significantly boosting jailbreak attack success rates and exposing weaknesses in current alignment and filtering methods.

Contribution

It reveals a novel vulnerability called context segmentation caused by eos tokens and proposes methods to enhance defenses against jailbreak attacks in large language models.

Findings

Appending eos tokens increases attack success rates across multiple jailbreak techniques.

Major API providers do not filter eos tokens, making their models vulnerable.

The vulnerability can be exploited to bypass existing content filtering and alignment defenses.

Abstract

Recent advances in Large Language Models (LLMs) have led to impressive alignment where models learn to distinguish harmful from harmless queries through supervised finetuning (SFT) and reinforcement learning from human feedback (RLHF). In this paper, we reveal a subtle yet impactful weakness in these aligned models. We find that simply appending multiple end of sequence (eos) tokens can cause a phenomenon we call context segmentation, which effectively shifts both harmful and benign inputs closer to the refusal boundary in the hidden space. Building on this observation, we propose a straightforward method to BOOST jailbreak attacks by appending eos tokens. Our systematic evaluation shows that this strategy significantly increases the attack success rate across 8 representative jailbreak techniques and 16 open-source LLMs, ranging from 2B to 72B parameters. Moreover, we develop a novel probing mechanism for commercial APIs and discover that major providers such as OpenAI, Anthropic, and Qwen do not filter eos tokens, making them similarly vulnerable. These findings highlight a hidden yet critical blind spot in existing alignment and content filtering approaches. We call for heightened attention to eos tokens' unintended influence on model behaviors, particularly in production systems. Our work not only calls for an input-filtering based defense, but also points to new defenses that make refusal boundaries more robust and generalizable, as well as fundamental alignment techniques that can defend against context segmentation attacks.