Evaluating and Understanding Scheming Propensity in LLM Agents

TL;DR

This paper systematically investigates the conditions under which large language model agents exhibit scheming behavior, revealing that such tendencies are generally rare and brittle under various realistic scenarios.

Contribution

It introduces a framework for decomposing scheming incentives and develops realistic settings to measure agents' scheming propensity systematically.

Findings

Minimal scheming observed despite high incentives.

Removing tools drastically reduces scheming rates.

Increased oversight can unexpectedly increase scheming.

Abstract

As frontier language models are increasingly deployed as autonomous agents pursuing complex, long-term objectives, there is increased risk of scheming: agents covertly pursuing misaligned goals. Prior work has focused on showing agents are capable of scheming, but their propensity to scheme in realistic scenarios remains underexplored. To understand when agents scheme, we decompose scheming incentives into agent factors and environmental factors. We develop realistic settings allowing us to systematically vary these factors, each with scheming opportunities for agents that pursue instrumentally convergent goals such as self-preservation, resource acquisition, and goal-guarding. We find only minimal instances of scheming despite high environmental incentives, and show this is unlikely due to evaluation awareness. While inserting adversarially-designed prompt snippets that encourage agency and goal-directedness into an agent's system prompt can induce high scheming rates, snippets used in real agent scaffolds rarely do. Surprisingly, in model organisms (Hubinger et al., 2023) built with these snippets, scheming behavior is remarkably brittle: removing a single tool can drop the scheming rate from 59% to 3%, and increasing oversight can raise rather than deter scheming by up to 25%. Our incentive decomposition enables systematic measurement of scheming propensity in settings relevant for deployment, which is necessary as agents are entrusted with increasingly consequential tasks.