ComplexTempQA:A 100m Dataset for Complex Temporal Question Answering

TL;DR

ComplexTempQA is a large-scale dataset with over 100 million questions designed to evaluate and improve temporal reasoning in question answering systems, covering diverse question types and requiring advanced reasoning skills.

Contribution

The paper introduces a novel, extensive dataset for complex temporal question answering, with detailed taxonomy and metadata to facilitate advanced temporal reasoning evaluation.

Findings

Dataset surpasses existing benchmarks in scale and complexity

Questions require multi-hop and temporal reasoning skills

Metadata enables detailed analysis of model performance

Abstract

We introduce \textsc{ComplexTempQA},\footnote{Dataset and code available at: https://github.com/DataScienceUIBK/ComplexTempQA} a large-scale dataset consisting of over 100 million question-answer pairs designed to tackle the challenges in temporal question answering. \textsc{ComplexTempQA} significantly surpasses existing benchmarks in scale and scope. Utilizing Wikipedia and Wikidata, the dataset covers questions spanning over two decades and offers an unmatched scale. We introduce a new taxonomy that categorizes questions as \textit{attributes}, \textit{comparisons}, and \textit{counting} questions, revolving around events, entities, and time periods, respectively. A standout feature of \textsc{ComplexTempQA} is the high complexity of its questions, which demand reasoning capabilities for answering such as across-time comparison, temporal aggregation, and multi-hop reasoning involving temporal event ordering and entity recognition. Additionally, each question is accompanied by detailed metadata, including specific time scopes, allowing for comprehensive evaluation of temporal reasoning abilities of large language models.