A multilingual hallucination benchmark: MultiWikiQHalluA

TL;DR

This paper introduces a multilingual hallucination benchmark using the MultiWikiQA dataset, evaluating hallucination rates across various languages and models, highlighting higher rates in lower-resource languages.

Contribution

It develops synthetic hallucination datasets for 306 languages and trains token-level classifiers for 30 European languages, enabling cross-lingual hallucination evaluation.

Findings

Higher hallucination rates in smaller models, especially in lower-resource languages.

Larger models like cogito-v1-preview outperform smaller models in reducing hallucinations.

Hallucination rates are notably higher for Icelandic and other low-resource languages.

Abstract

Most hallucination evaluations focus on English, leaving it unclear whether findings transfer to lower-resource languages. We investigate faithfulness hallucinations, defined as model-generated content that is fluent and plausible but diverges from the provided input or is internally inconsistent. Leveraging the multilingual MultiWikiQA dataset, we utilize the LettuceDetect framework to create synthetic hallucination datasets for 306 languages, from which we train token-level hallucination classifiers for 30 European languages. In this work, we present evaluations of model hallucinations on a selection of languages: English, Danish, German, and Icelandic. Using these classifiers, we evaluate the hallucination rates for Qwen3-0.6B, Qwen3-14B, Gemma-3-12B-IT, cogito-v1-preview-qwen-32B, and cogito-v1-preview-llama-70B. Our classifiers reveal notably higher hallucination rates for Qwen3-0.6B (up to 60\% of answers containing at least one hallucination, peaking in Icelandic) and generally lower rates for larger models, with cogito-v1-preview-qwen-32B and cogito-v1-preview-llama-70B performing best on most languages. Hallucination rates are consistently higher for lower-resource languages, particularly Icelandic.