Long-Range Transformer Architectures for Document Understanding

TL;DR

This paper introduces efficient long-range Transformer models for Document Understanding, capable of processing entire multi-page documents, with improved accuracy and relevance guidance through 2D relative attention bias.

Contribution

It presents two novel multi-modal long-range Transformer architectures for DU, incorporating 2D relative attention bias to enhance focus on relevant tokens while maintaining efficiency.

Findings

Long-range models outperform traditional models on multi-page documents.

2D relative attention bias improves token relevance detection.

Models maintain efficiency with only small performance trade-offs.

Abstract

Since their release, Transformers have revolutionized many fields from Natural Language Understanding to Computer Vision. Document Understanding (DU) was not left behind with first Transformer based models for DU dating from late 2019. However, the computational complexity of the self-attention operation limits their capabilities to small sequences. In this paper we explore multiple strategies to apply Transformer based models to long multi-page documents. We introduce 2 new multi-modal (text + layout) long-range models for DU. They are based on efficient implementations of Transformers for long sequences. Long-range models can process whole documents at once effectively and are less impaired by the document's length. We compare them to LayoutLM, a classical Transformer adapted for DU and pre-trained on millions of documents. We further propose 2D relative attention bias to guide self-attention towards relevant tokens without harming model efficiency. We observe improvements on multi-page business documents on Information Retrieval for a small performance cost on smaller sequences. Relative 2D attention revealed to be effective on dense text for both normal and long-range models.