Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders

TL;DR

This paper investigates why improved cross-lingual embedding alignment does not necessarily lead to better transfer performance, revealing the orthogonality of alignment and task objectives and providing practical fine-tuning guidelines.

Contribution

It demonstrates the disconnect between alignment quality and transfer success, analyzing gradient relationships and offering strategies for effective cross-lingual model fine-tuning.

Findings

Embedding distances are unreliable predictors of transfer performance.

Alignment and task gradients are often orthogonal, limiting transfer benefits.

Careful loss selection is crucial for effective cross-lingual transfer.

Abstract

Better cross-lingual alignment is often assumed to yield better cross-lingual transfer. However, explicit alignment techniques -- despite increasing embedding similarity -- frequently fail to improve token-level downstream performance. In this work, we show that this mismatch arises because alignment and downstream task objectives are largely orthogonal, and because the downstream benefits from alignment vary substantially across languages and task types. We analyze four XLM-R encoder models aligned on different language pairs and fine-tuned for either POS Tagging or Sentence Classification. Using representational analyses, including embedding distances, gradient similarities, and gradient magnitudes for both task and alignment losses, we find that: (1) embedding distances alone are unreliable predictors of improvements (or degradations) in task performance and (2) alignment and task gradients are often close to orthogonal, indicating that optimizing one objective may contribute little to optimizing the other. Taken together, our findings explain why ``better'' alignment often fails to translate into ``better'' cross-lingual transfer. Based on these insights, we provide practical guidelines for combining cross-lingual alignment with task-specific fine-tuning, highlighting the importance of careful loss selection.