ACL-Verbatim: hallucination-free question answering for research

TL;DR

This paper introduces ACL-Verbatim, an extractive question answering system for research papers that reduces hallucinations by directly mapping queries to text spans, with a new dataset and models evaluated on this benchmark.

Contribution

It presents a novel dataset and models for extractive QA in research papers, improving factual accuracy and reliability in AI-assisted research tools.

Findings

A 150M-parameter ModernBERT model achieves 53.6 F1 on the benchmark.

The dataset and pipeline enable training models with silver supervision.

VerbatimRAG effectively retrieves relevant text spans from research papers.

Abstract

Academic researchers need efficient and reliable methods for collecting high-quality information from trusted sources, but modern tools for AI-assisted research still suffer from the tendency of Large Language Models (LLMs) to produce factually inaccurate or nonsensical output, commonly referred to as hallucinations. We apply the extractive question answering system VerbatimRAG to research papers in the ACL Anthology, directly mapping user queries to verbatim text spans in retrieved documents. We contribute a novel ground truth dataset for the task of mapping user queries to relevant text spans in research papers, and use it to train and evaluate a variety of extractive models. Human annotation is performed by NLP researchers and is based on synthetic user queries generated using a custom pipeline based on the ScIRGen methodology, paired with chunks of research papers retrieved by VerbatimRAG. On this benchmark, a 150M-parameter ModernBERT token classifier trained on silver supervision from our pipeline achieves the best word-level F1 (53.6), ahead of the strongest evaluated LLM extractor (48.7).