Adapting Multilingual Embedding Models to Historical Luxembourgish

TL;DR

This paper explores adapting multilingual embedding models for effective semantic search in historical Luxembourgish, overcoming challenges posed by OCR noise and archaic spellings, and demonstrates significant accuracy improvements with new training data and adaptation techniques.

Contribution

It introduces a novel dataset and adaptation methods for multilingual embeddings to improve cross-lingual semantic search in historical Luxembourgish.

Findings

Existing models perform poorly on historical Luxembourgish.

Adaptation with new data improves accuracy across models.

Released models and data support future research.

Abstract

The growing volume of digitized historical texts requires effective semantic search using text embeddings. However, pre-trained multilingual models face challenges with historical content due to OCR noise and outdated spellings. This study examines multilingual embeddings for cross-lingual semantic search in historical Luxembourgish (LB), a low-resource language. We collect historical Luxembourgish news articles from various periods and use GPT-4o for sentence segmentation and translation, generating 20,000 parallel training sentences per language pair. Additionally, we create a semantic search (Historical LB Bitext Mining) evaluation set and find that existing models perform poorly on cross-lingual search for historical Luxembourgish. Using our historical and additional modern parallel training data, we adapt several multilingual embedding models through contrastive learning or knowledge distillation and increase accuracy significantly for all models. We release our adapted models and historical Luxembourgish-German/French/English bitexts to support further research.