Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages

TL;DR

This paper introduces a novel method that fuses all intermediate layers of multilingual encoders to enhance large language models' performance on low-resource languages without using multilingual training data.

Contribution

It proposes a new architecture with layer fusion strategies, including a Transformer Softmax, to improve multilingual understanding in LLMs trained solely on English data.

Findings

Significant performance improvements on LRL benchmarks.

Enhanced classification accuracy for Sinhala and Indic languages.

Overall increase in XNLI accuracy from 70.36% to 71.50%.

Abstract

Large Language Models (LLMs) excel in English, but their performance degrades significantly on low-resource languages (LRLs) due to English-centric training. While methods like LangBridge align LLMs with multilingual encoders such as the Massively Multilingual Text-to-Text Transfer Transformer (mT5), they typically use only the final encoder layer. We propose a novel architecture that fuses all intermediate layers, enriching the linguistic information passed to the LLM. Our approach features two strategies: (1) a Global Softmax weighting for overall layer importance, and (2) a Transformer Softmax model that learns token-specific weights. The fused representations are mapped into the LLM's embedding space, enabling it to process multilingual inputs. The model is trained only on English data, without using any parallel or multilingual data. Evaluated on XNLI, IndicXNLI, Sinhala News Classification, and Amazon Reviews, our Transformer Softmax model significantly outperforms the LangBridge baseline. We observe strong performance gains in LRLs, improving Sinhala classification accuracy from 71.66% to 75.86% and achieving clear improvements across Indic languages such as Tamil, Bengali, and Malayalam. These specific gains contribute to an overall boost in average XNLI accuracy from 70.36% to 71.50%. This approach offers a scalable, data-efficient path toward more capable and equitable multilingual LLMs.