WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval

TL;DR

WebFAQ is a comprehensive multilingual QA dataset derived from schema.org annotations, enabling improved dense retrieval models and high-quality bilingual corpora across over 1000 language pairs.

Contribution

The paper introduces WebFAQ, a large-scale, high-quality multilingual QA dataset and benchmarks, along with a method for generating high-quality bilingual corpora using automated bitext mining.

Findings

Fine-tuning XLM-RoBERTa on WebFAQ improves retrieval performance.

WebFAQ's datasets generalize well to other multilingual retrieval benchmarks.

High-quality bilingual corpora are created for over 1000 language pairs.

Abstract

We present WebFAQ, a large-scale collection of open-domain question answering datasets derived from FAQ-style schema.org annotations. In total, the data collection consists of 96 million natural question-answer (QA) pairs across 75 languages, including 47 million (49%) non-English samples. WebFAQ further serves as the foundation for 20 monolingual retrieval benchmarks with a total size of 11.2 million QA pairs (5.9 million non-English). These datasets are carefully curated through refined filtering and near-duplicate detection, yielding high-quality resources for training and evaluating multilingual dense retrieval models. To empirically confirm WebFAQ's efficacy, we use the collected QAs to fine-tune an in-domain pretrained XLM-RoBERTa model. Through this process of dataset-specific fine-tuning, the model achieves significant retrieval performance gains, which generalize - beyond WebFAQ - to other multilingual retrieval benchmarks evaluated in zero-shot setting. Last but not least, we utilize WebFAQ to construct a set of QA-aligned bilingual corpora spanning over 1000 language pairs using state-of-the-art bitext mining and automated LLM-assessed translation evaluation. Due to our advanced, automated method of bitext dataset generation, the resulting bilingual corpora demonstrate higher translation quality compared to similar datasets. WebFAQ and all associated resources are publicly available on GitHub and HuggingFace.