On the Limitations of Steering in Language Model Alignment

TL;DR

This paper evaluates the effectiveness and limitations of steering vectors for aligning language model behavior, highlighting their strengths in specific tasks and weaknesses in complex, general scenarios.

Contribution

It introduces a framework using transformer hook interventions and antonym-based vectors to assess steering vector limitations in LLMs.

Findings

Steering vectors work well for value alignment tasks.

They are less effective in complex, general-purpose alignment scenarios.

The paper provides a methodological foundation for future research.

Abstract

Steering vectors are a promising approach to aligning language model behavior at inference time. In this paper, we propose a framework to assess the limitations of steering vectors as alignment mechanisms. Using a framework of transformer hook interventions and antonym-based function vectors, we evaluate the role of prompt structure and context complexity in steering effectiveness. Our findings indicate that steering vectors are promising for specific alignment tasks, such as value alignment, but may not provide a robust foundation for general-purpose alignment in LLMs, particularly in complex scenarios. We establish a methodological foundation for future investigations into steering capabilities of reasoning models.