MFAQ: a Multilingual FAQ Dataset

TL;DR

This paper introduces the first large-scale multilingual FAQ dataset with 6 million pairs across 21 languages, evaluates various bi-encoder models, and highlights the benefits and challenges of multilingual FAQ retrieval.

Contribution

It provides the first publicly available multilingual FAQ dataset, benchmarks bi-encoder models on it, and analyzes cross-lingual transfer and model robustness.

Findings

Multilingual XLM-RoBERTa models outperform English-specific models.

Lower-resource languages benefit from multilingual training.

Models are sensitive to minor word changes, indicating brittleness.

Abstract

In this paper, we present the first multilingual FAQ dataset publicly available. We collected around 6M FAQ pairs from the web, in 21 different languages. Although this is significantly larger than existing FAQ retrieval datasets, it comes with its own challenges: duplication of content and uneven distribution of topics. We adopt a similar setup as Dense Passage Retrieval (DPR) and test various bi-encoders on this dataset. Our experiments reveal that a multilingual model based on XLM-RoBERTa achieves the best results, except for English. Lower resources languages seem to learn from one another as a multilingual model achieves a higher MRR than language-specific ones. Our qualitative analysis reveals the brittleness of the model on simple word changes. We publicly release our dataset, model and training script.