Hierarchical multimodal transformers for Multi-Page DocVQA

TL;DR

This paper extends Document Visual Question Answering to multi-page documents by creating a new dataset and proposing a hierarchical transformer model, Hi-VT5, that effectively processes long documents and offers explainability.

Contribution

Introduces MP-DocVQA dataset for multi-page DocVQA and proposes Hi-VT5, a hierarchical transformer model that handles long documents and provides explainability.

Findings

Hi-VT5 outperforms existing methods on MP-DocVQA.

The hierarchical approach effectively summarizes multi-page information.

The model can identify relevant pages as explanations.

Abstract

Document Visual Question Answering (DocVQA) refers to the task of answering questions from document images. Existing work on DocVQA only considers single-page documents. However, in real scenarios documents are mostly composed of multiple pages that should be processed altogether. In this work we extend DocVQA to the multi-page scenario. For that, we first create a new dataset, MP-DocVQA, where questions are posed over multi-page documents instead of single pages. Second, we propose a new hierarchical method, Hi-VT5, based on the T5 architecture, that overcomes the limitations of current methods to process long multi-page documents. The proposed method is based on a hierarchical transformer architecture where the encoder summarizes the most relevant information of every page and then, the decoder takes this summarized information to generate the final answer. Through extensive experimentation, we demonstrate that our method is able, in a single stage, to answer the questions and provide the page that contains the relevant information to find the answer, which can be used as a kind of explainability measure.