MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data

TL;DR

MultiHiertt is a new large-scale benchmark for complex numerical reasoning over multi-hierarchical tabular and textual data, particularly in financial reports, designed to challenge existing models and promote progress in hybrid data question answering.

Contribution

The paper introduces MultiHiertt, a novel benchmark with multi-hierarchical tables and textual data, and proposes MT2Net, a new model for complex numerical reasoning tasks.

Findings

Existing models perform poorly on MultiHiertt, lagging behind human performance.

MultiHiertt's complexity reveals limitations of current QA models in multi-step reasoning.

The dataset provides detailed annotations to facilitate research in numerical reasoning.

Abstract

Numerical reasoning over hybrid data containing both textual and tabular content (e.g., financial reports) has recently attracted much attention in the NLP community. However, existing question answering (QA) benchmarks over hybrid data only include a single flat table in each document and thus lack examples of multi-step numerical reasoning across multiple hierarchical tables. To facilitate data analytical progress, we construct a new large-scale benchmark, MultiHiertt, with QA pairs over Multi Hierarchical Tabular and Textual data. MultiHiertt is built from a wealth of financial reports and has the following unique characteristics: 1) each document contain multiple tables and longer unstructured texts; 2) most of tables contained are hierarchical; 3) the reasoning process required for each question is more complex and challenging than existing benchmarks; and 4) fine-grained annotations of reasoning processes and supporting facts are provided to reveal complex numerical reasoning. We further introduce a novel QA model termed MT2Net, which first applies facts retrieving to extract relevant supporting facts from both tables and text and then uses a reasoning module to perform symbolic reasoning over retrieved facts. We conduct comprehensive experiments on various baselines. The experimental results show that MultiHiertt presents a strong challenge for existing baselines whose results lag far behind the performance of human experts. The dataset and code are publicly available at https://github.com/psunlpgroup/MultiHiertt.