When Alignment Hurts: Decoupling Representational Spaces in Multilingual Models

TL;DR

This paper demonstrates that excessive alignment with high-resource languages in multilingual models can hinder low-resource language generation, and proposes a causal intervention method to improve performance by decoupling these representations.

Contribution

It introduces an online variational probing framework for decoupling high-resource language representations in LLMs, validated through Arabic dialect modeling.

Findings

Intervention improves dialectal translation quality by up to +4.9 chrF++

Decoupling high-resource language subspaces enhances low-resource language generation

Tradeoff observed in standard-language performance after intervention

Abstract

Alignment with high-resource standard languages is often assumed to aid the modeling of related low-resource varieties. We challenge this assumption by demonstrating that excessive representational entanglement with a dominant variety, such as Modern Standard Arabic (MSA) in relation to Arabic dialects, can actively hinder generative modeling. We present the first comprehensive causal study of this phenomenon by analyzing and directly intervening in the internal representation geometry of large language models (LLMs). Our key contribution is an online variational probing framework that continuously estimates the subspace of the standard variety during fine-tuning, enabling projection-based decoupling from this space. While our study uses Arabic as a case due to its unusually rich parallel resources across 25 dialects, the broader motivation is methodological: dialectal MT serves as a controlled proxy for generative tasks where comparable multi-variety corpora are unavailable. Across 25 dialects, our intervention improves generation quality by up to +4.9 chrF++ and +2.0 on average compared to standard fine-tuning, despite a measured tradeoff in standard-language performance. These results provide causal evidence that subspace dominance by high-resource varieties can restrict generative capacity for related varieties. More generally, we unify geometric and information-theoretic probing with subspace-level causal interventions, offering practical tools for improving generative modeling in closely related language families and, more broadly, for controlling representational allocation in multilingual and multi-domain LLMs. Code will be released.