Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification

TL;DR

This paper introduces RECV, a novel benchmark to evaluate LLMs' reasoning in claim verification, revealing strengths in deductive reasoning but challenges in abductive reasoning and the nuanced effects of rationale generation.

Contribution

The paper presents a new framework for breaking down claims into reasoning types and creates the first benchmark for assessing LLMs' reasoning in evidence-based claim verification.

Findings

LLMs perform well on deductive reasoning tasks.

LLMs struggle with abductive reasoning.

Rationale generation does not always improve LLM performance.

Abstract

Although LLMs have shown great performance on Mathematics and Coding related reasoning tasks, the reasoning capabilities of LLMs regarding other forms of reasoning are still an open problem. Here, we examine the issue of reasoning from the perspective of claim verification. We propose a framework designed to break down any claim paired with evidence into atomic reasoning types that are necessary for verification. We use this framework to create RECV, the first claim verification benchmark, incorporating real-world claims, to assess the deductive and abductive reasoning capabilities of LLMs. The benchmark comprises of three datasets, covering reasoning problems of increasing complexity. We evaluate three state-of-the-art proprietary LLMs under multiple prompt settings. Our results show that while LLMs can address deductive reasoning problems, they consistently fail in cases of abductive reasoning. Moreover, we observe that enhancing LLMs with rationale generation is not always beneficial. Nonetheless, we find that generated rationales are semantically similar to those provided by humans, especially in deductive reasoning cases.