Improving Multilingual Language Models by Aligning Representations through Steering

TL;DR

This paper introduces a lightweight representation steering method that improves multilingual performance of large language models by aligning internal representations, achieving results comparable to translation systems with fewer resources.

Contribution

The paper presents a novel, efficient intervention technique using representation steering to enhance multilingual capabilities of LLMs, outperforming many existing methods.

Findings

Outperforms most baseline methods in multilingual tasks

Achieves translation-quality performance with fewer resources

Complementary to supervised fine-tuning, enhancing internal representations

Abstract

This paper investigates how Large Language Models (LLMs) represent non-English tokens -- a question that remains underexplored despite recent progress. We propose a lightweight intervention method using representation steering, where a learned vector is added to the residual stream at a single model layer to enhance multilingual performance. Through extensive experiments across seven competitive baselines -- including prompt optimization, supervised fine-tuning (SFT), in-context learning, cross-lingual transfer, and translation-based methods-we show that our approach consistently outperforms most alternatives. In particular, it achieves performance on par with production-grade translation systems while requiring far fewer resources. We further explore the complementarity between our method and SFT, demonstrating that steering offers a direct, efficient way to realign internal representations. These findings underscore the potential of activation-level interventions as a powerful tool for improving the multilingual capabilities of LLMs.