"Why is this misleading?": Detecting News Headline Hallucinations with Explanations

TL;DR

This paper introduces ExHalder, a framework that detects hallucinated news headlines by leveraging natural language inference data and provides human-readable explanations, addressing the challenge of dataset scarcity and improving detection accuracy.

Contribution

The paper presents ExHalder, a novel approach that adapts inference knowledge to news headlines and generates explanations, enhancing hallucination detection in news summarization.

Findings

ExHalder achieves high accuracy in detecting hallucinated headlines.

The model provides human-readable explanations for its predictions.

Extensive experiments validate the effectiveness across multiple datasets.

Abstract

Automatic headline generation enables users to comprehend ongoing news events promptly and has recently become an important task in web mining and natural language processing. With the growing need for news headline generation, we argue that the hallucination issue, namely the generated headlines being not supported by the original news stories, is a critical challenge for the deployment of this feature in web-scale systems Meanwhile, due to the infrequency of hallucination cases and the requirement of careful reading for raters to reach the correct consensus, it is difficult to acquire a large dataset for training a model to detect such hallucinations through human curation. In this work, we present a new framework named ExHalder to address this challenge for headline hallucination detection. ExHalder adapts the knowledge from public natural language inference datasets into the news domain and learns to generate natural language sentences to explain the hallucination detection results. To evaluate the model performance, we carefully collect a dataset with more than six thousand labeled <article, headline> pairs. Extensive experiments on this dataset and another six public ones demonstrate that ExHalder can identify hallucinated headlines accurately and justifies its predictions with human-readable natural language explanations.