Rethinking Cross-lingual Alignment: Balancing Transfer and Cultural Erasure in Multilingual LLMs

TL;DR

This paper examines the trade-off between factual transfer and cultural localization in multilingual LLMs, proposing a new method to better balance these objectives by steering internal representations at different model layers.

Contribution

It introduces the transfer-localization plane framework and Surgical Steering, a novel inference-time technique to disentangle and balance transfer and cultural localization.

Findings

CLA approaches improve factual transfer but reduce cultural localization

Universal transfer and cultural knowledge are steerable at different model layers

Surgical Steering improves the balance between transfer and localization

Abstract

Cross-lingual alignment (CLA) aims to align multilingual representations, enabling Large Language Models (LLMs) to seamlessly transfer knowledge across languages. While intuitive, we hypothesize, this pursuit of representational convergence can inadvertently cause "cultural erasure", the functional loss of providing culturally-situated responses that should diverge based on the query language. In this work, we systematically analyze this trade-off by introducing a holistic evaluation framework, the transfer-localization plane, which quantifies both desirable knowledge transfer and undesirable cultural erasure. Using this framework, we re-evaluate recent CLA approaches and find that they consistently improve factual transfer at the direct cost of cultural localization across all six languages studied. Our investigation into the internal representations of these models reveals a key insight: universal factual transfer and culturally-specific knowledge are optimally steerable at different model layers. Based on this finding, we propose Surgical Steering, a novel inference-time method that disentangles these two objectives. By applying targeted activation steering to distinct layers, our approach achieves a better balance between the two competing dimensions, effectively overcoming the limitations of current alignment techniques.