Modelling the semantics of text in complex document layouts using graph transformer networks

TL;DR

This paper introduces a graph transformer-based model that creates unified semantic representations of text spans across complex document layouts, enabling cross-type semantic retrieval and capturing meaningful semantic information.

Contribution

The novel approach models structured document text as a graph, allowing unified semantic embeddings regardless of content type, bridging the gap between different machine learning techniques.

Findings

Effective retrieval of semantically similar information across documents

Generated embeddings capture useful semantic information

Model works across various content types in complex documents

Abstract

Representing structured text from complex documents typically calls for different machine learning techniques, such as language models for paragraphs and convolutional neural networks (CNNs) for table extraction, which prohibits drawing links between text spans from different content types. In this article we propose a model that approximates the human reading pattern of a document and outputs a unique semantic representation for every text span irrespective of the content type they are found in. We base our architecture on a graph representation of the structured text, and we demonstrate that not only can we retrieve semantically similar information across documents but also that the embedding space we generate captures useful semantic information, similar to language models that work only on text sequences.