EMULATE: A Multi-Agent Framework for Determining the Veracity of Atomic Claims by Emulating Human Actions

TL;DR

EMULATE is a multi-agent system that mimics human actions to verify atomic claims more effectively, improving fact-checking accuracy by emulating human-like evidence gathering and evaluation processes.

Contribution

This paper introduces EMULATE, a novel multi-agent framework that better emulates human actions for claim verification, outperforming previous retrieval and classification methods.

Findings

Significant accuracy improvements over prior methods

Effective evidence ranking and evaluation by specialized agents

Robust performance across multiple benchmark datasets

Abstract

Determining the veracity of atomic claims is an imperative component of many recently proposed fact-checking systems. Many approaches tackle this problem by first retrieving evidence by querying a search engine and then performing classification by providing the evidence set and atomic claim to a large language model, but this process deviates from what a human would do in order to perform the task. Recent work attempted to address this issue by proposing iterative evidence retrieval, allowing for evidence to be collected several times and only when necessary. Continuing along this line of research, we propose a novel claim verification system, called EMULATE, which is designed to better emulate human actions through the use of a multi-agent framework where each agent performs a small part of the larger task, such as ranking search results according to predefined criteria or evaluating webpage content. Extensive experiments on several benchmarks show clear improvements over prior work, demonstrating the efficacy of our new multi-agent framework.