Multi-hop Evidence Pursuit Meets the Web: Team Papelo at FEVER 2024

TL;DR

This paper presents a multi-hop evidence pursuit system combining large language models and search engines to verify claims on the web, improving accuracy and evidence adequacy in the FEVER 2024 shared task.

Contribution

It introduces a novel iterative question generation and evidence retrieval approach that enhances claim verification performance over traditional single-step methods.

Findings

Achieved 0.477 AVeriTeC score on the FEVER 2024 test set.

Improved label accuracy by 0.045 over all-at-once question generation strategy.

Highlighted the effectiveness of various design choices like question formulation and metadata inclusion.

Abstract

Separating disinformation from fact on the web has long challenged both the search and the reasoning powers of humans. We show that the reasoning power of large language models (LLMs) and the retrieval power of modern search engines can be combined to automate this process and explainably verify claims. We integrate LLMs and search under a multi-hop evidence pursuit strategy. This strategy generates an initial question based on an input claim using a sequence to sequence model, searches and formulates an answer to the question, and iteratively generates follow-up questions to pursue the evidence that is missing using an LLM. We demonstrate our system on the FEVER 2024 (AVeriTeC) shared task. Compared to a strategy of generating all the questions at once, our method obtains .045 higher label accuracy and .155 higher AVeriTeC score (evaluating the adequacy of the evidence). Through ablations, we show the importance of various design choices, such as the question generation method, medium-sized context, reasoning with one document at a time, adding metadata, paraphrasing, reducing the problem to two classes, and reconsidering the final verdict. Our submitted system achieves .510 AVeriTeC score on the dev set and .477 AVeriTeC score on the test set.