ANHALTEN: Cross-Lingual Transfer for German Token-Level Reference-Free Hallucination Detection

TL;DR

This paper introduces ANHALTEN, a German dataset for token-level hallucination detection, and demonstrates that few-shot cross-lingual transfer approaches are effective, facilitating research in languages with limited data.

Contribution

It is the first work to explore cross-lingual transfer for token-level reference-free hallucination detection and provides a new dataset for German.

Findings

Larger context length improves detection accuracy.

Few-shot transfer is highly effective with minimal data.

Cross-lingual transfer enables effective hallucination detection in German.

Abstract

Research on token-level reference-free hallucination detection has predominantly focused on English, primarily due to the scarcity of robust datasets in other languages. This has hindered systematic investigations into the effectiveness of cross-lingual transfer for this important NLP application. To address this gap, we introduce ANHALTEN, a new evaluation dataset that extends the English hallucination detection dataset to German. To the best of our knowledge, this is the first work that explores cross-lingual transfer for token-level reference-free hallucination detection. ANHALTEN contains gold annotations in German that are parallel (i.e., directly comparable to the original English instances). We benchmark several prominent cross-lingual transfer approaches, demonstrating that larger context length leads to better hallucination detection in German, even without succeeding context. Importantly, we show that the sample-efficient few-shot transfer is the most effective approach in most setups. This highlights the practical benefits of minimal annotation effort in the target language for reference-free hallucination detection. Aiming to catalyze future research on cross-lingual token-level reference-free hallucination detection, we make ANHALTEN publicly available: https://github.com/janekh24/anhalten