User-Centric Evidence Ranking for Attribution and Fact Verification

TL;DR

This paper introduces Evidence Ranking, a new task to prioritize evidence presentation for fact verification, aiming to reduce user effort and improve verification accuracy using novel ranking strategies and LLMs.

Contribution

It proposes a new evidence ranking task, develops an evaluation framework, and demonstrates the effectiveness of incremental ranking and LLM-based methods in improving verification efficiency.

Findings

Incremental ranking better captures complementary evidence.

LLM-based methods outperform shallow baselines.

Evidence ranking reduces reading effort and enhances verification.

Abstract

Attribution and fact verification are critical challenges in natural language processing for assessing information reliability. While automated systems and Large Language Models (LLMs) aim to retrieve and select concise evidence to support or refute claims, they often present users with either insufficient or overly redundant information, leading to inefficient and error-prone verification. To address this, we propose Evidence Ranking, a novel task that prioritizes presenting sufficient information as early as possible in a ranked list. This minimizes user reading effort while still making all available evidence accessible for sequential verification. We compare two approaches for the new ranking task: one-shot ranking and incremental ranking. We introduce a new evaluation framework, inspired by information retrieval metrics, and construct a unified benchmark by aggregating existing fact verification datasets. Extensive experiments with diverse models show that incremental ranking strategies better capture complementary evidence and that LLM-based methods outperform shallower baselines, while still facing challenges in balancing sufficiency and redundancy. Compared to evidence selection, we conduct a controlled user study and demonstrate that evidence ranking both reduces reading effort and improves verification. This work provides a foundational step toward more interpretable, efficient, and user-aligned information verification systems.