Steerability of Instrumental-Convergence Tendencies in LLMs

TL;DR

This paper investigates how the ability to steer AI systems toward desired behaviors is affected by capability growth, highlighting a safety-security trade-off and demonstrating that anti-instrumental prompts can significantly reduce instrumental convergence in large language models.

Contribution

It introduces the concept of instrumental convergence steerability, distinguishes safety and security aspects, and empirically shows how anti-instrumental prompts can mitigate convergence tendencies in LLMs.

Findings

Anti-instrumental prompts sharply reduce convergence rates.

Larger models show lower convergence under anti-instrumental prompts.

Steerability trade-offs pose safety and security challenges.

Abstract

We examine two properties of AI systems: capability (what a system can do) and steerability (how reliably one can shift behavior toward intended outcomes). A central question is whether capability growth reduces steerability and risks control collapse. We also distinguish between authorized steerability (builders reliably reaching intended behaviors) and unauthorized steerability (attackers eliciting disallowed behaviors). This distinction highlights a fundamental safety--security dilemma of AI models: safety requires high steerability to enforce control (e.g., stop/refuse), while security requires low steerability for malicious actors to elicit harmful behaviors. This tension presents a significant challenge for open-weight models, which currently exhibit high steerability via common techniques like fine-tuning or adversarial attacks. Using Qwen3 and InstrumentalEval, we find that a short anti-instrumental prompt suffix sharply reduces the measured convergence rate (e.g., shutdown avoidance, self-replication). For Qwen3-30B Instruct, the convergence rate drops from 81.69% under a pro-instrumental suffix to 2.82% under an anti-instrumental suffix. Under anti-instrumental prompting, larger aligned models show lower convergence rates than smaller ones (Instruct: 2.82% vs. 4.23%; Thinking: 4.23% vs. 9.86%). Code is available at github.com/j-hoscilowicz/instrumental_steering.