Correcting Hallucinations in News Summaries: Exploration of Self-Correcting LLM Methods with External Knowledge

TL;DR

This paper explores self-correcting large language models to reduce hallucinations in news summaries by leveraging external evidence and multi-turn verification, revealing practical benefits of search snippets and few-shot prompts.

Contribution

It applies and analyzes two state-of-the-art self-correcting methods for news summarization, highlighting their effectiveness and practical insights in this domain.

Findings

Search engine snippets improve correction accuracy

Few-shot prompts enhance system performance

G-Eval aligns well with human judgments

Abstract

While large language models (LLMs) have shown remarkable capabilities to generate coherent text, they suffer from the issue of hallucinations -- factually inaccurate statements. Among numerous approaches to tackle hallucinations, especially promising are the self-correcting methods. They leverage the multi-turn nature of LLMs to iteratively generate verification questions inquiring additional evidence, answer them with internal or external knowledge, and use that to refine the original response with the new corrections. These methods have been explored for encyclopedic generation, but less so for domains like news summarization. In this work, we investigate two state-of-the-art self-correcting systems by applying them to correct hallucinated summaries using evidence from three search engines. We analyze the results and provide insights into systems' performance, revealing interesting practical findings on the benefits of search engine snippets and few-shot prompts, as well as high alignment of G-Eval and human evaluation.