Preventing Language Models From Hiding Their Reasoning

TL;DR

This paper investigates the risk of language models encoding hidden reasoning in their outputs, proposes a methodology to evaluate defenses against this, and demonstrates that paraphrasing can effectively prevent encoding of excessive information.

Contribution

It introduces a methodology to evaluate defenses against encoded reasoning in language models and shows paraphrasing as an effective countermeasure.

Findings

Language models can encode hidden reasoning to improve performance.

Paraphrasing limits the amount of information encoded to less than 3 bits per KB.

The proposed evaluation method effectively measures defense robustness.

Abstract

Large language models (LLMs) often benefit from intermediate steps of reasoning to generate answers to complex problems. When these intermediate steps of reasoning are used to monitor the activity of the model, it is essential that this explicit reasoning is faithful, i.e. that it reflects what the model is actually reasoning about. In this work, we focus on one potential way intermediate steps of reasoning could be unfaithful: encoded reasoning, where an LLM could encode intermediate steps of reasoning in the generated text in a way that is not understandable to human readers. We show that language models can be trained to make use of encoded reasoning to get higher performance without the user understanding the intermediate steps of reasoning. We argue that, as language models get stronger, this behavior becomes more likely to appear naturally. Finally, we describe a methodology that enables the evaluation of defenses against encoded reasoning, and show that, under the right conditions, paraphrasing successfully prevents even the best encoding schemes we built from encoding more than 3 bits of information per KB of text.