Zero-shot and Few-shot Learning with Instruction-following LLMs for Claim Matching in Automated Fact-checking

TL;DR

This paper explores zero-shot and few-shot learning with instruction-following large language models for claim matching in automated fact-checking, introducing a new dataset and evaluating different prompt strategies.

Contribution

It is the first to apply instruction-following LLMs to claim matching, proposing a new dataset and a pipeline for handling texts of varying lengths.

Findings

LLMs can effectively perform claim matching using zero-shot and few-shot prompts.

Prompt design significantly impacts model performance.

Claim matching can leverage tasks like natural language inference and paraphrase detection.

Abstract

The claim matching (CM) task can benefit an automated fact-checking pipeline by putting together claims that can be resolved with the same fact-check. In this work, we are the first to explore zero-shot and few-shot learning approaches to the task. We consider CM as a binary classification task and experiment with a set of instruction-following large language models (GPT-3.5-turbo, Gemini-1.5-flash, Mistral-7B-Instruct, and Llama-3-8B-Instruct), investigating prompt templates. We introduce a new CM dataset, ClaimMatch, which will be released upon acceptance. We put LLMs to the test in the CM task and find that it can be tackled by leveraging more mature yet similar tasks such as natural language inference or paraphrase detection. We also propose a pipeline for CM, which we evaluate on texts of different lengths.