OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs

TL;DR

OpenFactCheck introduces a comprehensive framework for building, benchmarking, and evaluating the factual accuracy of claims and LLM outputs, addressing the need for standardized assessment tools in open-domain fact-checking.

Contribution

It provides a unified platform with modules for customizing fact-checkers, evaluating LLM factuality, and verifying verification results, facilitating fair comparison and progress.

Findings

OpenFactCheck enables customizable fact-checking systems.

It offers a unified evaluation framework for LLM factuality.

The platform supports reliable verification with human-annotated datasets.

Abstract

The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. Difficulties lie in assessing the factuality of free-form responses in open domains. Also, different papers use disparate evaluation benchmarks and measurements, which renders them hard to compare and hampers future progress. To mitigate these issues, we propose OpenFactCheck, a unified framework for building customized automatic fact-checking systems, benchmarking their accuracy, evaluating factuality of LLMs, and verifying claims in a document. OpenFactCheck consists of three modules: (i) CUSTCHECKER allows users to easily customize an automatic fact-checker and verify the factual correctness of documents and claims, (ii) LLMEVAL, a unified evaluation framework assesses LLM's factuality ability from various perspectives fairly, and (iii) CHECKEREVAL is an extensible solution for gauging the reliability of automatic fact-checkers' verification results using human-annotated datasets. Data and code are publicly available at https://github.com/yuxiaw/openfactcheck.