Multilingual Fine-Grained News Headline Hallucination Detection

TL;DR

This paper presents a new multilingual, fine-grained dataset for detecting hallucinations in news headlines, along with methods to improve detection using fine-tuning and large language models, addressing a gap in multilingual and nuanced hallucination analysis.

Contribution

It introduces the first multilingual, fine-grained hallucination detection dataset with expert annotations and proposes novel techniques to enhance large language models' detection capabilities.

Findings

Supervised fine-tuning demonstrates dataset's utility and challenges.

Large language models' in-context learning can be improved with proposed techniques.

The dataset enables nuanced analysis of hallucination types across languages.

Abstract

The popularity of automated news headline generation has surged with advancements in pre-trained language models. However, these models often suffer from the ``hallucination'' problem, where the generated headline is not fully supported by its source article. Efforts to address this issue have predominantly focused on English, using over-simplistic classification schemes that overlook nuanced hallucination types. In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand pairs in 5 languages, each annotated with detailed hallucination types by experts. We conduct extensive experiments on this dataset under two settings. First, we implement several supervised fine-tuning approaches as preparatory solutions and demonstrate this dataset's challenges and utilities. Second, we test various large language models' in-context learning abilities and propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance in terms of the example-F1 metric. We release this dataset to foster further research in multilingual, fine-grained headline hallucination detection.