SynClaimEval: A Framework for Evaluating the Utility of Synthetic Data in Long-Context Claim Verification

TL;DR

This paper introduces SynClaimEval, a framework for assessing how synthetic data can improve long-context claim verification in language models, focusing on input characteristics, synthesis logic, and explanation quality.

Contribution

It presents a comprehensive evaluation framework for synthetic data utility in long-context claim verification, highlighting its benefits for model performance and explainability.

Findings

Synthetic data improves verification accuracy when augmenting datasets.

Synthesis enhances explanation quality even without score improvements.

Long-context synthesis benefits instruction-tuned models.

Abstract

Large Language Models (LLMs) with extended context windows promise direct reasoning over long documents, reducing the need for chunking or retrieval. Constructing annotated resources for training and evaluation, however, remains costly. Synthetic data offers a scalable alternative, and we introduce SynClaimEval, a framework for evaluating synthetic data utility in long-context claim verification -- a task central to hallucination detection and fact-checking. Our framework examines three dimensions: (i) input characteristics, by varying context length and testing generalization to out-of-domain benchmarks; (ii) synthesis logic, by controlling claim complexity and error type variation; and (iii) explanation quality, measuring the degree to which model explanations provide evidence consistent with predictions. Experiments across benchmarks show that long-context synthesis can improve verification in base instruction-tuned models, particularly when augmenting existing human-written datasets. Moreover, synthesis enhances explanation quality, even when verification scores do not improve, underscoring its potential to strengthen both performance and explainability.