Evaluating Language Model Reasoning about Confidential Information

TL;DR

This paper introduces PasswordEval, a benchmark to assess whether language models can correctly identify authorized requests involving confidential information, revealing current models' struggles and potential safety risks in high-stakes applications.

Contribution

The paper develops a new benchmark, PasswordEval, to evaluate language models' ability to handle confidential info and demonstrates their limitations in reasoning and safety in high-stakes contexts.

Findings

Models struggle with password verification tasks.

Reasoning traces often leak confidential information.

Performance does not improve with increased reasoning complexity.

Abstract

As language models are increasingly deployed as autonomous agents in high-stakes settings, ensuring that they reliably follow user-defined rules has become a critical safety concern. To this end, we study whether language models exhibit contextual robustness, or the capability to adhere to context-dependent safety specifications. For this analysis, we develop a benchmark (PasswordEval) that measures whether language models can correctly determine when a user request is authorized (i.e., with a correct password). We find that current open- and closed-source models struggle with this seemingly simple task, and that, perhaps surprisingly, reasoning capabilities do not generally improve performance. In fact, we find that reasoning traces frequently leak confidential information, which calls into question whether reasoning traces should be exposed to users in such applications. We also scale the difficulty of our evaluation along multiple axes: (i) by adding adversarial user pressure through various jailbreaking strategies, and (ii) through longer multi-turn conversations where password verification is more challenging. Overall, our results suggest that current frontier models are not well-suited to handling confidential information, and that reasoning capabilities may need to be trained in a different manner to make them safer for release in high-stakes settings.

Figures