Multimodal Pre-training Based on Graph Attention Network for Document Understanding

TL;DR

GraphDoc is a multimodal graph attention-based model that pre-trains on text, layout, and image data to improve document understanding across diverse formats and layouts.

Contribution

It introduces a novel graph attention mechanism incorporating multimodal features for pre-training on unlabeled documents, enhancing document understanding performance.

Findings

Achieves state-of-the-art results on public datasets.

Effectively models contextual relationships in documents.

Utilizes only 320k unlabeled documents for pre-training.

Abstract

Document intelligence as a relatively new research topic supports many business applications. Its main task is to automatically read, understand, and analyze documents. However, due to the diversity of formats (invoices, reports, forms, etc.) and layouts in documents, it is difficult to make machines understand documents. In this paper, we present the GraphDoc, a multimodal graph attention-based model for various document understanding tasks. GraphDoc is pre-trained in a multimodal framework by utilizing text, layout, and image information simultaneously. In a document, a text block relies heavily on its surrounding contexts, accordingly we inject the graph structure into the attention mechanism to form a graph attention layer so that each input node can only attend to its neighborhoods. The input nodes of each graph attention layer are composed of textual, visual, and positional features from semantically meaningful regions in a document image. We do the multimodal feature fusion of each node by the gate fusion layer. The contextualization between each node is modeled by the graph attention layer. GraphDoc learns a generic representation from only 320k unlabeled documents via the Masked Sentence Modeling task. Extensive experimental results on the publicly available datasets show that GraphDoc achieves state-of-the-art performance, which demonstrates the effectiveness of our proposed method. The code is available at https://github.com/ZZR8066/GraphDoc.