Cross-Prompt Encoder for Low-Performing Languages

TL;DR

This paper introduces the Cross-Prompt Encoder (XPE), a lightweight architecture that enhances transfer learning for low-performing languages in multilingual models, using multi-source training and hybrid prompt mechanisms.

Contribution

It proposes the XPE architecture and a Dual Soft Prompt mechanism, improving transferability and performance on low-resource languages in multilingual models.

Findings

XPE improves performance on low-performing languages.

Hybrid prompts enhance adaptability across languages.

XPE is most effective for low-resource language transfer.

Abstract

Soft prompts have emerged as a powerful alternative to adapters in parameter-efficient fine-tuning (PEFT), enabling large language models (LLMs) to adapt to downstream tasks without architectural changes or parameter updates. While prior work has focused on stabilizing training via parameter interaction in small neural prompt encoders, their broader potential for transfer across languages remains unexplored. In this paper, we demonstrate that a prompt encoder can play a central role in improving performance on low-performing languages - those that achieve poor accuracy even under full-model fine-tuning. We investigate a lightweight encoder paired with multi-source training on typologically diverse languages. We call this architecture-training combination the Cross-Prompt Encoder (XPE), and show that it advances the capture of abstract, transferable patterns across languages. To complement XPE, we propose a Dual Soft Prompt mechanism that combines an encoder-based prompt with a directly trained standard soft prompt. This hybrid design proves especially effective for target languages that benefit from both broadly shared structure and language-specific alignment. Text classification experiments with a transformer encoder (XLM-R) on the SIB-200 benchmark reveal a consistent trade-off: XPE is most effective for low-performing languages, while hybrid variants offer broader adaptability across multilingual settings.