HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims

TL;DR

HerO is a system that uses only publicly available large language models to verify real-world claims, achieving high accuracy in a fact-checking shared task by enhancing evidence retrieval and claim verification.

Contribution

This work introduces a fully open LLM-based pipeline for fact-checking that improves evidence retrieval and claim verification without proprietary models.

Findings

Achieved 2nd place on AVeriTeC leaderboard with a score of 0.57.

Demonstrated the effectiveness of open LLMs in real-world claim verification.

Published code for reproducibility and future research.

Abstract

To tackle the AVeriTeC shared task hosted by the FEVER-24, we introduce a system that only employs publicly available large language models (LLMs) for each step of automated fact-checking, dubbed the Herd of Open LLMs for verifying real-world claims (HerO). For evidence retrieval, a language model is used to enhance a query by generating hypothetical fact-checking documents. We prompt pretrained and fine-tuned LLMs for question generation and veracity prediction by crafting prompts with retrieved in-context samples. HerO achieved 2nd place on the leaderboard with the AVeriTeC score of 0.57, suggesting the potential of open LLMs for verifying real-world claims. For future research, we make our code publicly available at https://github.com/ssu-humane/HerO.