Bootstrapping Embeddings for Low Resource Languages

TL;DR

This paper explores methods to generate synthetic data for training language embeddings in low-resource languages, demonstrating that adapter composition and XL-LoRA significantly improve performance over existing approaches.

Contribution

The paper introduces two novel strategies, adapter composition and XL-LoRA, for generating synthetic data to enhance embedding models in low-resource languages.

Findings

Adapter composition and XL-LoRA outperform in-context learning.

Synthetic data approaches yield strong performance gains.

Methods provide scalable solutions for low-resource language NLP.

Abstract

Embedding models are crucial to modern NLP. However, the creation of the most effective models relies on carefully constructed supervised finetuning data. For high resource languages, such as English, such datasets are readily available. However, for hundreds of other languages, they are simply non-existent. We investigate whether the advent of large language models can help to bridge this gap. We test three different strategies for generating synthetic triplet data used to optimise embedding models. These include in-context learning as well as two novel approaches, leveraging adapter composition and cross lingual finetuning of the LLM generator (XL-LoRA) respectively. We find that while in-context learning still falls short of strong non-synthetic baselines, adapter composition and XL-LoRA yield strong performance gains across a wide array of tasks and languages, offering a clear, scalable pathway to producing performant embedding models for a wide variety of languages.