Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models

TL;DR

This paper investigates how model interventions, specifically finding experts, can enhance cross-lingual alignment in multilingual language models, leading to improved retrieval performance without extensive fine-tuning.

Contribution

It demonstrates that targeted activation manipulations can improve cross-lingual alignment and retrieval accuracy in mLLMs, offering a data-efficient alternative to fine-tuning.

Findings

Activation manipulation enhances cross-lingual alignment.

Improved retrieval accuracy up to 2x in top-1 performance.

Identified neurons responsible for language-specific representations.

Abstract

Aligned representations across languages is a desired property in multilingual large language models (mLLMs), as alignment can improve performance in cross-lingual tasks. Typically alignment requires fine-tuning a model, which is computationally expensive, and sizable language data, which often may not be available. A data-efficient alternative to fine-tuning is model interventions -- a method for manipulating model activations to steer generation into the desired direction. We analyze the effect of a popular intervention (finding experts) on the alignment of cross-lingual representations in mLLMs. We identify the neurons to manipulate for a given language and introspect the embedding space of mLLMs pre- and post-manipulation. We show that modifying the mLLM's activations changes its embedding space such that cross-lingual alignment is enhanced. Further, we show that the changes to the embedding space translate into improved downstream performance on retrieval tasks, with up to 2x improvements in top-1 accuracy on cross-lingual retrieval.