FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoning

TL;DR

This paper introduces FG-PRM, a model that detects and mitigates specific types of hallucinations in LLMs during mathematical reasoning, improving accuracy and interpretability through fine-grained supervision and automated data generation.

Contribution

The paper presents a new taxonomy of hallucinations, a fine-grained detection and mitigation model, and an automated data generation method for training, advancing the reliability of LLM reasoning.

Findings

FG-PRM outperforms existing methods in hallucination detection

Significantly improves LLM performance on GSM8K and MATH benchmarks

Automated data generation reduces manual labeling effort

Abstract

Hallucinations in large language models (LLMs) pose significant challenges in tasks requiring complex multi-step reasoning, such as mathematical problem-solving. Existing approaches primarily detect the presence of hallucinations but lack a nuanced understanding of their types and manifestations. In this paper, we first introduce a comprehensive taxonomy that categorizes the common hallucinations in mathematical reasoning tasks into six types. We then propose FG-PRM (Fine-Grained Process Reward Model), an augmented model designed to detect and mitigate hallucinations in a fine-grained, step-level manner. To address the limitations of manually labeling training data, we propose an automated method for generating fine-grained hallucination data using LLMs. Our FG-PRM demonstrates superior performance across two key tasks: 1) Fine-grained hallucination detection: classifying hallucination types for each reasoning step; and 2) Verification: ranking multiple LLM-generated outputs to select the most accurate solution. Our experiments show that FG-PRM excels in fine-grained hallucination detection and substantially boosts the performance of LLMs on GSM8K and MATH benchmarks. These results highlight the benefits of fine-grained supervision in enhancing the reliability and interpretability of LLM reasoning processes.