ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment

TL;DR

This paper introduces ODESteer, an ODE-based activation steering framework for LLM alignment that unifies theoretical understanding and demonstrates empirical improvements over existing methods.

Contribution

It provides a unified ODE-based theoretical framework for activation steering and introduces ODESteer, a novel method guided by barrier functions for improved LLM alignment.

Findings

Achieves up to 5.7% improvement on TruthfulQA

Demonstrates consistent empirical gains over state-of-the-art methods

Validates the ODE framework as a principled approach to activation steering

Abstract

Activation steering, or representation engineering, offers a lightweight approach to align large language models (LLMs) by manipulating their internal activations at inference time. However, current methods suffer from two key limitations: (i) the lack of a unified theoretical framework for guiding the design of steering directions, and (ii) an over-reliance on one-step steering that fail to capture complex patterns of activation distributions. In this work, we propose a unified ordinary differential equations (ODEs)-based theoretical framework for activation steering in LLM alignment. We show that conventional activation addition can be interpreted as a first-order approximation to the solution of an ODE. Based on this ODE perspective, identifying a steering direction becomes equivalent to designing a barrier function from control theory. Derived from this framework, we introduce ODESteer, a kind of ODE-based steering guided by barrier functions, which shows empirical advancement in LLM alignment. ODESteer identifies steering directions by defining the barrier function as the log-density ratio between positive and negative activations, and employs it to construct an ODE for multi-step and adaptive steering. Compared to state-of-the-art activation steering methods, ODESteer achieves consistent empirical improvements on diverse LLM alignment benchmarks, a notable $5.7\%$ improvement over TruthfulQA, $2.5\%$ over UltraFeedback, and $2.4\%$ over RealToxicityPrompts. Our work establishes a principled new view of activation steering in LLM alignment by unifying its theoretical foundations via ODEs, and validating it empirically through the proposed ODESteer method.