Improving Instruction-Following in Language Models through Activation Steering

TL;DR

This paper introduces activation steering, a method that uses instruction-specific activation vectors to improve language models' adherence to constraints and instructions during inference, enhancing control and transferability.

Contribution

The paper presents a novel activation steering technique that enables modular, inference-time control of language models based on instruction-specific activation vectors, including compositional and transfer capabilities.

Findings

Activation vectors improve model adherence to constraints.

Steering enables control without explicit instructions.

Transferability enhances base model performance.

Abstract

The ability to follow instructions is crucial for numerous real-world applications of language models. In pursuit of deeper insights and more powerful capabilities, we derive instruction-specific vector representations from language models and use them to steer models accordingly. These vectors are computed as the difference in activations between inputs with and without instructions, enabling a modular approach to activation steering. We demonstrate how this method can enhance model adherence to constraints such as output format, length, and word inclusion, providing inference-time control over instruction following. Our experiments across four models demonstrate how we can use the activation vectors to guide models to follow constraints even without explicit instructions and to enhance performance when instructions are present. Additionally, we explore the compositionality of activation steering, successfully applying multiple instructions simultaneously. Finally, we demonstrate that steering vectors computed on instruction-tuned models can transfer to improve base models. Our findings demonstrate that activation steering offers a practical and scalable approach for fine-grained control in language generation. Our code and data are available at https://github.com/microsoft/llm-steer-instruct.