AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task

TL;DR

This paper presents a simple Retrieval-Augmented Generation approach for automated fact-checking, achieving third place in the AVeriTeC shared task by leveraging large language models and detailed modules for evidence retrieval and label generation.

Contribution

The authors introduce a RAG-based fact-checking system with novel features like MMR-reranking and confidence estimation, and provide a detailed implementation and evaluation on the AVeriTeC dataset.

Findings

GPT-4o was the most suitable model for the pipeline.

Llama 3.1 70B is a promising open-source alternative.

Error analysis revealed noise and ambiguity as key challenges.

Abstract

This paper describes our $3^{rd}$ place submission in the AVeriTeC shared task in which we attempted to address the challenge of fact-checking with evidence retrieved in the wild using a simple scheme of Retrieval-Augmented Generation (RAG) designed for the task, leveraging the predictive power of Large Language Models. We release our codebase and explain its two modules - the Retriever and the Evidence & Label generator - in detail, justifying their features such as MMR-reranking and Likert-scale confidence estimation. We evaluate our solution on AVeriTeC dev and test set and interpret the results, picking the GPT-4o as the most appropriate model for our pipeline at the time of our publication, with Llama 3.1 70B being a promising open-source alternative. We perform an empirical error analysis to see that faults in our predictions often coincide with noise in the data or ambiguous fact-checks, provoking further research and data augmentation.