SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers

TL;DR

SciDQA is a rigorously curated dataset of 2,937 scientific QA pairs sourced from peer reviews and author responses, designed to challenge LLMs in deep, multi-document scientific comprehension involving reasoning across various complex materials.

Contribution

The paper introduces SciDQA, a new scientific reading comprehension dataset with high-quality, naturally derived questions that require multi-document reasoning and are sourced from expert peer reviews and author answers.

Findings

LLMs show significant performance gaps on SciDQA

Questions require reasoning across figures, tables, and supplementary materials

Dataset enables evaluation of deep scientific understanding in LLMs

Abstract

Scientific literature is typically dense, requiring significant background knowledge and deep comprehension for effective engagement. We introduce SciDQA, a new dataset for reading comprehension that challenges LLMs for a deep understanding of scientific articles, consisting of 2,937 QA pairs. Unlike other scientific QA datasets, SciDQA sources questions from peer reviews by domain experts and answers by paper authors, ensuring a thorough examination of the literature. We enhance the dataset's quality through a process that carefully filters out lower quality questions, decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. Questions in SciDQA necessitate reasoning across figures, tables, equations, appendices, and supplementary materials, and require multi-document reasoning. We evaluate several open-source and proprietary LLMs across various configurations to explore their capabilities in generating relevant and factual responses. Our comprehensive evaluation, based on metrics for surface-level similarity and LLM judgements, highlights notable performance discrepancies. SciDQA represents a rigorously curated, naturally derived scientific QA dataset, designed to facilitate research on complex scientific text understanding.