Enhancing Visually-Rich Document Understanding via Layout Structure Modeling

TL;DR

This paper introduces GraphLayoutLM, a new model that incorporates layout structure graphs into document understanding, significantly improving performance by modeling spatial relationships between text elements.

Contribution

The paper presents GraphLayoutLM, a novel approach that integrates layout structure graphs and layout-aware self-attention to enhance visually-rich document understanding.

Findings

Achieves state-of-the-art results on FUNSD, XFUND, and CORD datasets.

Both graph reordering and layout-aware attention are crucial for optimal performance.

Significant improvement over existing models by incorporating layout information.

Abstract

In recent years, the use of multi-modal pre-trained Transformers has led to significant advancements in visually-rich document understanding. However, existing models have mainly focused on features such as text and vision while neglecting the importance of layout relationship between text nodes. In this paper, we propose GraphLayoutLM, a novel document understanding model that leverages the modeling of layout structure graph to inject document layout knowledge into the model. GraphLayoutLM utilizes a graph reordering algorithm to adjust the text sequence based on the graph structure. Additionally, our model uses a layout-aware multi-head self-attention layer to learn document layout knowledge. The proposed model enables the understanding of the spatial arrangement of text elements, improving document comprehension. We evaluate our model on various benchmarks, including FUNSD, XFUND and CORD, and achieve state-of-the-art results among these datasets. Our experimental results demonstrate that our proposed method provides a significant improvement over existing approaches and showcases the importance of incorporating layout information into document understanding models. We also conduct an ablation study to investigate the contribution of each component of our model. The results show that both the graph reordering algorithm and the layout-aware multi-head self-attention layer play a crucial role in achieving the best performance.