Less is More: Adapting Text Embeddings for Low-Resource Languages with Small Scale Noisy Synthetic Data

TL;DR

This paper demonstrates that for low-resource languages, small-scale noisy synthetic data can effectively adapt text embeddings, achieving high performance with minimal data and challenging the need for large, pristine datasets.

Contribution

Introduces a cost-effective adaptation method using limited noisy synthetic data for low-resource languages, showing strong results and robustness over traditional large-scale approaches.

Findings

10,000 noisy synthetic pairs improve retrieval by 20%+

Performance matches models trained on 1 million examples

Increasing data quality or diversity yields limited gains

Abstract

Low-resource languages (LRLs) often lack high-quality, large-scale datasets for training effective text embedding models, hindering their application in tasks like retrieval-augmented generation (RAG) and semantic search. In this work, we challenge the prevailing assumption that effective semantic alignment requires massive datasets or pristine, human-verified translations. Focusing on Armenian (an LRL with a unique script), we introduce a cost-effective adaptation strategy using small scale noisy synthetic data generated by translating English Reddit title-body pairs with open-weights models. We establish a comprehensive evaluation benchmark comprising existing datasets, translated data, and a manually curated dataset. Our experiments reveal a surprising "Less is More" phenomenon: fine-tuning a multilingual encoder (mE5) on just 10,000 noisy synthetic pairs yields 11-12\% average improvements across the benchmark with a 20\%+ relative improvement in retrieval performance, matching the performance of models trained on ~1 million examples. Furthermore, we demonstrate that neither increasing data scale, improving translation quality via state-of-the-art LLMs, nor diversifying data domains yields significant gains over this minimal baseline. We validate the generalizability of these findings on another LRL with a unique script. Our results suggest that semantic alignment for LRLs saturates early and is highly robust to noise, democratizing high-performance embedding creation for resource-constrained communities. We release the model, data, and the benchmark at https://metric-ai-lab.github.io/less-is-more-embeddings/ to facilitate further research.