Face4RAG: Factual Consistency Evaluation for Retrieval Augmented Generation in Chinese

TL;DR

This paper introduces Face4RAG, a comprehensive benchmark for evaluating factual consistency in retrieval-augmented generation across various LLMs, and proposes L-Face4RAG, a new method that significantly improves detection of logical fallacies and factual errors.

Contribution

The paper presents the first LLM-independent FCE benchmark and a novel method, L-Face4RAG, that enhances factual inconsistency detection in RAG systems.

Findings

Existing FCE methods fail to detect logical fallacies.

L-Face4RAG outperforms previous methods across multiple tasks.

The benchmark and method are publicly available.

Abstract

The prevailing issue of factual inconsistency errors in conventional Retrieval Augmented Generation (RAG) motivates the study of Factual Consistency Evaluation (FCE). Despite the various FCE methods proposed earlier, these methods are evaluated on datasets generated by specific Large Language Models (LLMs). Without a comprehensive benchmark, it remains unexplored how these FCE methods perform on other LLMs with different error distributions or even unseen error types, as these methods may fail to detect the error types generated by other LLMs. To fill this gap, in this paper, we propose the first comprehensive FCE benchmark \emph{Face4RAG} for RAG independent of the underlying LLM. Our benchmark consists of a synthetic dataset built upon a carefully designed typology for factuality inconsistency error and a real-world dataset constructed from six commonly used LLMs, enabling evaluation of FCE methods on specific error types or real-world error distributions. On the proposed benchmark, we discover the failure of existing FCE methods to detect the logical fallacy, which refers to a mismatch of logic structures between the answer and the retrieved reference. To fix this issue, we further propose a new method called \emph{L-Face4RAG} with two novel designs of logic-preserving answer decomposition and fact-logic FCE. Extensive experiments show L-Face4RAG substantially outperforms previous methods for factual inconsistency detection on a wide range of tasks, notably beyond the RAG task from which it is originally motivated. Both the benchmark and our proposed method are publicly available.\footnote{\url{https://huggingface.co/datasets/yq27/Face4RAG}\label{link_face4rag}}