Improving Factuality with Explicit Working Memory

TL;DR

The paper introduces EWE, a novel explicit working memory system that improves factual accuracy in long-form text generation by real-time feedback and retrieval, outperforming existing methods on multiple datasets.

Contribution

EWE is a new approach that integrates real-time feedback into working memory to enhance factuality in language models, addressing limitations of previous retrieval-augmented methods.

Findings

EWE increases the VeriScore by 2-6 points across datasets.

Memory update rules and retrieval quality significantly impact performance.

EWE maintains helpfulness while improving factual accuracy.

Abstract

Large language models can generate factually inaccurate content, a problem known as hallucination. Recent works have built upon retrieved-augmented generation to improve factuality through iterative prompting but these methods are limited by the traditional RAG design. To address these challenges, we introduce EWE (Explicit Working Memory), a novel approach that enhances factuality in long-form text generation by integrating a working memory that receives real-time feedback from external resources. The memory is refreshed based on online fact-checking and retrieval feedback, allowing EWE to rectify false claims during the generation process and ensure more accurate and reliable outputs. Our experiments demonstrate that Ewe outperforms strong baselines on four fact-seeking long-form generation datasets, increasing the factuality metric, VeriScore, by 2 to 6 points absolute without sacrificing the helpfulness of the responses. Further analysis reveals that the design of rules for memory updates, configurations of memory units, and the quality of the retrieval datastore are crucial factors for influencing model performance.