Multilingual Self-Taught Faithfulness Evaluators
Carlo Alfano, Aymen Al Marjani, Zeno Jonke, Amin Mantrach, Saab Mansour, Marcello Federico

TL;DR
This paper introduces a multilingual faithfulness evaluation framework for LLMs that learns from synthetic data and cross-lingual transfer, outperforming existing methods in multilingual settings.
Contribution
It proposes a novel self-taught evaluation approach that operates across languages without requiring extensive labeled data, leveraging synthetic data and transfer learning.
Findings
The framework improves faithfulness evaluation accuracy across multiple languages.
Cross-lingual transfer correlates with language capabilities of LLMs.
Outperforms state-of-the-art English-only and translation-based evaluators.
Abstract
The growing use of large language models (LLMs) has increased the need for automatic evaluation systems, particularly to address the challenge of information hallucination. Although existing faithfulness evaluation approaches have shown promise, they are predominantly English-focused and often require expensive human-labeled training data for fine-tuning specialized models. As LLMs see increased adoption in multilingual contexts, there is a need for accurate faithfulness evaluators that can operate across languages without extensive labeled data. This paper presents Self-Taught Evaluators for Multilingual Faithfulness, a framework that learns exclusively from synthetic multilingual summarization data while leveraging cross-lingual transfer learning. Through experiments comparing language-specific and mixed-language fine-tuning approaches, we demonstrate a consistent relationship between…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
