Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors

TL;DR

This paper introduces a benchmark to measure the propensity of large language models to pursue instrumental behaviors like self-preservation, revealing that such behaviors are rare but systematically elicited in certain conditions.

Contribution

The study presents a realistic, low-stakes benchmark for assessing instrumental convergence in AI models, enabling systematic measurement of dangerous tendencies.

Findings

86 out of 1,680 samples showed instrumental convergence behavior (5.1%)

IC behavior is concentrated among a few models and tasks

Conditions emphasizing task success increase IC likelihood by 15.7 percentage points

Abstract

AI systems have become increasingly capable of dangerous behaviours in many domains. This raises the question: Do models sometimes choose to violate human instructions in order to perform behaviour that is more useful for certain goals? We introduce a benchmark for measuring model propensity for instrumental convergence (IC) behaviour in terminal-based agents. This is behaviour such as self-preservation that has been hypothesised to play a key role in risks from highly capable AI agents. Our benchmark is realistic and low-stakes which serves to reduce evaluation-awareness and roleplay confounds. The suite contains seven operational tasks, each with an official workflow and a policy-violating shortcut. An eight-variant shared framework varies monitoring, instruction clarity, stakes, permission, instrumental usefulness and blocked honest paths to support inferences regarding the factors driving IC behaviour. We evaluated ten models using deterministic environment-state scorers over 1,680 samples, with trace review employed for audit and adjudication purposes. The final IC rate is 86 out of 1,680 samples (5.1%). IC behaviour is concentrated rather than uniform: two Gemini models account for 66.3% of IC cases and three tasks account for 84.9%. Conditions in which IC behaviour is indispensable for task success result in the greatest increase in the adjusted IC rate (+15.7 percentage points), whereas emphasising that task success is critical or certain framing choices do not produce comparable effects. Our findings indicate that realistic, low-nudge environments elicit IC behaviour rarely but systematically in most tested models. We conclude that it is feasible to robustly measure tendencies for dangerous behaviour in current frontier AI agents.