OverThink: Slowdown Attacks on Reasoning LLMs

TL;DR

OverThink is a novel attack that exploits reasoning language models by injecting benign decoy problems, significantly increasing inference latency and costs without compromising answer correctness, with implications for security and efficiency.

Contribution

The paper introduces OverThink, a new slowdown attack on reasoning LLMs that leverages decoy reasoning problems to increase inference overhead without detection.

Findings

OverThink significantly increases reasoning token usage and inference latency.

The attack transfers across different models and modalities.

Defenses can mitigate but not fully prevent the slowdown effects.

Abstract

Most flagship language models generate explicit reasoning chains, enabling inference-time scaling. However, producing these reasoning chains increases token usage (i.e., reasoning tokens), which in turn increases latency and costs. Our OverThink attack increases overhead for applications that rely on reasoning language models (RLMs) and external context by forcing them to spend substantially more reasoning tokens while still producing contextually correct answers. An adversary mounts an attack by injecting decoy reasoning problems into public content that is consumed by RLM at inference time. Because our decoys (e.g., Markov decision processes, Sudokus, etc.) are benign, they evade safety filters. We evaluate OverThink on both closed-source and open-source reasoning models across the FreshQA, SQuAD, and MuSR datasets. We also explore the attack in multi-modal settings by creating images that cause excessive reasoning. We show that the resulting slowdown transfers across models. Finally, we explore both LLM-based and systems-level defenses, and discuss the societal, financial, and energy implications of the OverThink attacks.