Iterative Hierarchical Attention for Answering Complex Questions over Long Documents

TL;DR

The paper introduces DocHopper, a hierarchical attention model that navigates long documents to answer complex, multi-hop questions efficiently, achieving state-of-the-art results on multiple datasets.

Contribution

It presents a novel hierarchical attention mechanism that enables multi-step navigation through long documents for question answering.

Findings

Achieves state-of-the-art results on three QA datasets.

Runs 3-10 times faster than baseline models.

Effectively handles long, complex documents for multi-hop reasoning.

Abstract

We propose a new model, DocHopper, that iteratively attends to different parts of long, hierarchically structured documents to answer complex questions. Similar to multi-hop question-answering (QA) systems, at each step, DocHopper uses a query $q$ to attend to information from a document, combines this ``retrieved'' information with $q$ to produce the next query. However, in contrast to most previous multi-hop QA systems, DocHopper is able to ``retrieve'' either short passages or long sections of the document, thus emulating a multi-step process of ``navigating'' through a long document to answer a question. To enable this novel behavior, DocHopper does not combine document information with $q$ by concatenating text to the text of $q$, but by combining a compact neural representation of $q$ with a compact neural representation of a hierarchical part of the document, which can potentially be quite large. We experiment with DocHopper on four different QA tasks that require reading long and complex documents to answer multi-hop questions, and show that DocHopper achieves state-of-the-art results on three of the datasets. Additionally, DocHopper is efficient at inference time, being 3--10 times faster than the baselines.