Language Steering for Multilingual In-Context Learning

TL;DR

This paper introduces language vectors as activation offsets to improve multilingual in-context learning, enabling models to switch languages effectively without retraining.

Contribution

It proposes a simple method to steer language in multilingual models using activation offsets, demonstrating consistent improvements across multiple tasks and languages.

Findings

Language vectors improve multilingual task performance.

Vectors encode interpretable linguistic structure.

Language identity occupies structured directions in activation space.

Abstract

If large language models operate in a universal semantic space, then switching between languages should require only a simple activation offset. To test this, we take multilingual in-context learning as a case study, where few-shot demonstrations are provided in English but the test query is in a target language. We propose language vectors, computed as the mean activation difference between parallel source and target language examples at a particular layer, and added as an offset to hidden states at inference time to shift the model's internal representations toward the target language. We evaluate our method across three multilingual tasks spanning 19 languages and three models. Our results show consistent improvements on multilingual in-context learning over baselines across all tasks and languages tested, demonstrating that a simple activation offset is sufficient to redirect a model's language mode without any parameter updates. Beyond performance, the vectors encode interpretable linguistic structure, with closely related languages forming tight clusters and vectors transferring across tasks, suggesting that language identity occupies separable and structured directions in a model's activation space.