Languages are Modalities: Cross-Lingual Alignment via Encoder Injection

TL;DR

This paper introduces LLINK, a novel method for improving cross-lingual performance of instruction-tuned LLMs on low-resource languages by aligning sentence embeddings without retraining the decoder or changing the tokenizer.

Contribution

LLINK is a compute-efficient approach that conditions a frozen decoder using latent language injection, enhancing cross-lingual alignment and retrieval for low-resource languages.

Findings

Significant improvement in bilingual retrieval performance.

Achieved 81.3% preference over the base model in Q&A evaluations.

Reduced tokenization inflation and strengthened cross-lingual alignment.

Abstract

Instruction-tuned Large Language Models (LLMs) underperform on low resource, non-Latin scripts due to tokenizer fragmentation and weak cross-lingual coupling. We present LLINK (Latent Language Injection for Non-English Knowledge), a compute efficient language-as-modality method that conditions an instruction-tuned decoder without changing the tokenizer or retraining the decoder. First, we align sentence embeddings from a frozen multilingual encoder to the decoder's latent embedding space at a reserved position via a lightweight contrastive projector. Second, the vector is expanded into K soft slots and trained with minimal adapters so the frozen decoder consumes the signal. LLINK substantially improves bilingual retrieval and achieves 81.3% preference over the base model and 63.6% over direct fine-tuning in LLM-judged Q&A evaluations. We further find that improvements can be attributed to reduced tokenization inflation and a stronger cross lingual alignment, despite the model having residual weaknesses in numeric fidelity. Treating low resource languages as a modality offers a practical path to stronger cross-lingual alignment in lightweight LLMs.