Skim-Attention: Learning to Focus via Document Layout

TL;DR

Skim-Attention introduces a layout-aware attention mechanism that improves document understanding efficiency by focusing on document structure, reducing computational costs, and enhancing performance of pre-trained models on long and structured documents.

Contribution

The paper proposes Skim-Attention, a novel layout-based attention mechanism that leverages document structure to improve efficiency and performance in document understanding tasks.

Findings

Skim-Attention achieves lower perplexity than previous methods.

It is more computationally efficient than traditional multimodal models.

Skim-Attention enhances the performance of existing language models when used as a mask.

Abstract

Transformer-based pre-training techniques of text and layout have proven effective in a number of document understanding tasks. Despite this success, multimodal pre-training models suffer from very high computational and memory costs. Motivated by human reading strategies, this paper presents Skim-Attention, a new attention mechanism that takes advantage of the structure of the document and its layout. Skim-Attention only attends to the 2-dimensional position of the words in a document. Our experiments show that Skim-Attention obtains a lower perplexity than prior works, while being more computationally efficient. Skim-Attention can be further combined with long-range Transformers to efficiently process long documents. We also show how Skim-Attention can be used off-the-shelf as a mask for any Pre-trained Language Model, allowing to improve their performance while restricting attention. Finally, we show the emergence of a document structure representation in Skim-Attention.