Non-linear Interventions on Large Language Models

TL;DR

This paper introduces a novel intervention framework for large language models that extends to non-linear features, enabling more precise control and understanding of model internal representations.

Contribution

It presents a general formulation and learning procedure for non-linear interventions, surpassing linear methods in steering LLM behaviors.

Findings

Outperforms linear baselines in refusal bypass steering

Enables intervention on implicit features without direct output signatures

Validates the framework on non-linear feature manipulation

Abstract

Intervention is one of the most representative and widely used methods for understanding the internal representations of large language models (LLMs). However, existing intervention methods are confined to linear interventions grounded in the Linear Representation Hypothesis, leaving features encoded along non-linear manifolds beyond their reach. In this work, we introduce a general formulation of intervention that extends naturally to non-linearly represented features, together with a learning procedure that further enables intervention on implicit features lacking a direct output signature. We validate our framework on refusal bypass steering, where it steers the model more precisely than linear baselines by intervening on a non-linear feature governing refusal.