Early Signs of Steganographic Capabilities in Frontier LLMs

TL;DR

This paper investigates the early steganographic abilities of frontier large language models, finding they can encode messages with additional tools and perform basic encoded reasoning, but currently cannot reliably evade monitoring.

Contribution

It provides an empirical evaluation of LLM steganography, revealing nascent capabilities and identifying conditions that enable encoding and reasoning within generated text.

Findings

Models struggle to hide messages without detection under standard conditions.

Additional affordances like scratchpads enable encoding messages.

Models show initial signs of performing encoded reasoning.

Abstract

Monitoring Large Language Model (LLM) outputs is crucial for mitigating risks from misuse and misalignment. However, LLMs could evade monitoring through steganography: Encoding hidden information within seemingly benign generations. In this paper, we evaluate the steganography capabilities in frontier LLMs to better understand the risk they pose. We focus on two types of steganography: passing encoded messages and performing encoded reasoning. We find that current models are unable to encode short messages in their outputs without a monitor noticing under standard affordances. They can succeed, however, if given additional affordances like using an unmonitored scratchpad and coordinating on what encoding scheme to use. We additionally find early signs that models can perform basic encoded reasoning in a simple state-tracking problem. This includes some ability to reason with their own and pre-defined schemes, including encoding schemes such as Hexadecimal. Despite this, they can rarely hide reasoning subtly within a cover task to fool a monitor. Overall, our results indicate that current LLMs exhibit nascent steganographic capabilities. While these capabilities are likely insufficient to bypass well-designed monitors at present, this could change in the future.