ASTRA-QA: A Benchmark for Abstract Question Answering over Documents

TL;DR

ASTRA-QA introduces a comprehensive benchmark for evaluating abstract question answering over documents, emphasizing answer coverage, unsupported content detection, and retrieval robustness.

Contribution

The paper presents ASTRA-QA, a new benchmark with detailed annotations and scalable evaluation metrics for abstract QA over long and multi-document texts.

Findings

RAG methods show varying coverage and hallucination issues on ASTRA-QA.

ASTRA-QA enables diagnostics for retrieval scope and answer support.

Benchmark supports scalable, annotation-based evaluation of abstract QA.

Abstract

Document-based question answering (QA) increasingly includes abstract questions that require synthesizing scattered information from long documents or across multiple documents into coherent answers. However, this setting is still poorly supported by existing benchmarks and evaluation methods, which often lack stable abstract references or rely on coarse similarity metrics and unstable head-to-head comparisons. To alleviate this issue, we introduce ASTRA-QA, a benchmark for AbSTRAct Question Answering over documents. ASTRA-QA contains 869 QA instances over academic papers and news documents, covering five abstract question types and three controlled retrieval scopes. Each instance is equipped with explicit evaluation annotations, including answer topic sets, curated unsupported topics, and aligned evidence. Building on these annotations, ASTRA-QA assesses whether answers cover required key points and avoid unsupported content by directly scoring topic coverage and curated unsupported content, enabling scalable evaluation without exhaustive head-to-head comparisons. Experiments with representative Retrieval-Augmented Generation (RAG) methods spanning vanilla, graph-based, and hierarchical retrieval settings show that ASTRA-QA provides reference-grounded diagnostics for coverage, hallucination, and retrieval-scope robustness. Our dataset and code are available at https://xinyangsally.github.io/astra-benchmark.