Are Large Language Models Good Fact Checkers: A Preliminary Study

TL;DR

This study evaluates the potential of large language models in fact-checking tasks, highlighting their strengths and limitations, especially in handling Chinese verification and complex pipelines, to guide future improvements.

Contribution

It provides a systematic evaluation of LLMs in fact-checking, comparing their performance with smaller models and identifying key challenges and areas for enhancement.

Findings

LLMs perform competitively in most fact-checking scenarios.

Challenges exist in Chinese fact verification and full pipeline handling.

Hallucinations and language inconsistencies limit LLM effectiveness.

Abstract

Recently, Large Language Models (LLMs) have drawn significant attention due to their outstanding reasoning capabilities and extensive knowledge repository, positioning them as superior in handling various natural language processing tasks compared to other language models. In this paper, we present a preliminary investigation into the potential of LLMs in fact-checking. This study aims to comprehensively evaluate various LLMs in tackling specific fact-checking subtasks, systematically evaluating their capabilities, and conducting a comparative analysis of their performance against pre-trained and state-of-the-art low-parameter models. Experiments demonstrate that LLMs achieve competitive performance compared to other small models in most scenarios. However, they encounter challenges in effectively handling Chinese fact verification and the entirety of the fact-checking pipeline due to language inconsistencies and hallucinations. These findings underscore the need for further exploration and research to enhance the proficiency of LLMs as reliable fact-checkers, unveiling the potential capability of LLMs and the possible challenges in fact-checking tasks.