PFME: A Modular Approach for Fine-grained Hallucination Detection and Editing of Large Language Models

TL;DR

This paper introduces PFME, a modular framework that detects and edits fine-grained hallucinations in large language models by integrating real-time fact retrieval and sentence-level editing, significantly improving accuracy over existing methods.

Contribution

The paper presents PFME, a novel framework combining fact retrieval and sentence-level editing for fine-grained hallucination detection and correction in LLMs.

Findings

PFME outperforms existing methods in hallucination detection.

PFME improves FActScore on multiple datasets.

Performance gains are significant with external knowledge assistance.

Abstract

Large Language Models (LLMs) excel in fluency but risk producing inaccurate content, called "hallucinations." This paper outlines a standardized process for categorizing fine-grained hallucination types and proposes an innovative framework--the Progressive Fine-grained Model Editor (PFME)--specifically designed to detect and correct fine-grained hallucinations in LLMs. PFME consists of two collaborative modules: the Real-time Fact Retrieval Module and the Fine-grained Hallucination Detection and Editing Module. The former identifies key entities in the document and retrieves the latest factual evidence from credible sources. The latter further segments the document into sentence-level text and, based on relevant evidence and previously edited context, identifies, locates, and edits each sentence's hallucination type. Experimental results on FavaBench and FActScore demonstrate that PFME outperforms existing methods in fine-grained hallucination detection tasks. Particularly, when using the Llama3-8B-Instruct model, PFME's performance in fine-grained hallucination detection with external knowledge assistance improves by 8.7 percentage points (pp) compared to ChatGPT. In editing tasks, PFME further enhances the FActScore of FActScore-Alpaca13B and FActScore-ChatGPT datasets, increasing by 16.2pp and 4.6pp, respectively.