LargePiG: Your Large Language Model is Secretly a Pointer Generator

TL;DR

This paper introduces LargePiG, a novel method that transforms large language models into pointer-generators to reduce hallucinations in query generation, improving factual accuracy and relevance.

Contribution

The paper presents a training-free, model-agnostic approach to separate content from form in LLM-generated queries, leveraging inherent attention weights to mitigate hallucinations.

Findings

LargePiG outperforms existing methods on document and video datasets.

LargePiG reduces hallucinations in vision-language models.

Improves factuality and accuracy in question-answering tasks.

Abstract

Recent research on query generation has focused on using Large Language Models (LLMs), which despite bringing state-of-the-art performance, also introduce issues with hallucinations in the generated queries. In this work, we introduce relevance hallucination and factuality hallucination as a new typology for hallucination problems brought by query generation based on LLMs. We propose an effective way to separate content from form in LLM-generated queries, which preserves the factual knowledge extracted and integrated from the inputs and compiles the syntactic structure, including function words, using the powerful linguistic capabilities of the LLM. Specifically, we introduce a model-agnostic and training-free method that turns the Large Language Model into a Pointer-Generator (LargePiG), where the pointer attention distribution leverages the LLM's inherent attention weights, and the copy probability is derived from the difference between the vocabulary distribution of the model's high layers and the last layer. To validate the effectiveness of LargePiG, we constructed two datasets for assessing the hallucination problems in query generation, covering both document and video scenarios. Empirical studies on various LLMs demonstrated the superiority of LargePiG on both datasets. Additional experiments also verified that LargePiG could reduce hallucination in large vision language models and improve the accuracy of document-based question-answering and factuality evaluation tasks.