MArgE: Meshing Argumentative Evidence from Multiple Large Language Models for Justifiable Claim Verification

TL;DR

MArgE introduces a structured argumentative framework for claim verification using multiple LLMs, enhancing justification transparency and significantly outperforming unstructured multi-LLM approaches and individual models.

Contribution

The paper presents MArgE, a novel framework that constructs argument trees from multiple LLMs to improve claim verification and provide transparent justifications.

Findings

MArgE outperforms single LLMs and prior debate methods.

Structured argument trees enhance verification accuracy.

Formal argumentative reasoning improves multi-LLM evidence integration.

Abstract

Leveraging outputs from multiple large language models (LLMs) is emerging as a method for harnessing their power across a wide range of tasks while mitigating their capacity for making errors, e.g., hallucinations. However, current approaches to combining insights from multiple LLMs often involve unstructured interactions (e.g., free debate), resulting in model generations that are not faithfully justifiable. In this work, we introduce MArgE, a novel framework to provide formal structure to the evidence from each LLM, in the form of a tree of extracted arguments, for the task of claim verification. We use a variant of Argumentative LLMs (ArgLLMs), i.e. LLMs driven by frameworks and semantics from the field of computational argumentation, to construct structured argument trees for given claims. This process creates an inspectable pathway from the initial arguments to the final claim verification decisions, providing a faithful justification thereof. We show experimentally that MArgE can significantly outperform single LLMs, including three open-source models (4B to 8B parameters), GPT-4o-mini and existing ArgLLMs, as well as prior methods for unstructured multi-LLM debates. We thus demonstrate the advantages of incorporating formal, argumentative reasoning mechanisms when combining multiple LLM outputs.