StrucTexT: Structured Text Understanding with Multi-Modal Transformers

TL;DR

StrucTexT introduces a unified multi-modal transformer framework for structured text understanding in visually rich documents, effectively handling entity labeling and linking through novel pre-training strategies and outperforming existing methods.

Contribution

The paper presents a flexible, unified transformer-based framework with a new pre-training strategy for efficient structured text understanding at multiple levels.

Findings

Outperforms state-of-the-art on FUNSD, SROIE, and EPHOIE datasets.

Effectively handles both entity labeling and linking tasks.

Utilizes multi-modal information across text, image, and layout.

Abstract

Structured text understanding on Visually Rich Documents (VRDs) is a crucial part of Document Intelligence. Due to the complexity of content and layout in VRDs, structured text understanding has been a challenging task. Most existing studies decoupled this problem into two sub-tasks: entity labeling and entity linking, which require an entire understanding of the context of documents at both token and segment levels. However, little work has been concerned with the solutions that efficiently extract the structured data from different levels. This paper proposes a unified framework named StrucTexT, which is flexible and effective for handling both sub-tasks. Specifically, based on the transformer, we introduce a segment-token aligned encoder to deal with the entity labeling and entity linking tasks at different levels of granularity. Moreover, we design a novel pre-training strategy with three self-supervised tasks to learn a richer representation. StrucTexT uses the existing Masked Visual Language Modeling task and the new Sentence Length Prediction and Paired Boxes Direction tasks to incorporate the multi-modal information across text, image, and layout. We evaluate our method for structured text understanding at segment-level and token-level and show it outperforms the state-of-the-art counterparts with significantly superior performance on the FUNSD, SROIE, and EPHOIE datasets.