Global memory transformer for processing long documents

TL;DR

This paper investigates the use of general memory slots added to transformer inputs to improve handling of long documents, demonstrating enhanced performance on masked language modeling and question answering tasks.

Contribution

It provides a systematic study of memory slots rule application in transformers, showing their effectiveness in processing long chunks compared to baseline models.

Findings

Memory slots improve masked language modeling performance.

Using compressed input chunks slightly degrades performance.

Memory-augmented transformer outperforms baseline T5 on long document tasks.

Abstract

Transformer variants dominate the state-of-the-art in different natural language processing tasks such as translation, reading comprehension and summarization. Our paper is more directed to use general memory slots added to the inputs and studying the results of adding these slots. This paper is a go on study of general memory slots rule that were added to the input of the proposed model in previous work. We have two main tasks;1) pretraining task using masked language modeling and b) fine tuning task using HotpotQA . This study aims to verify the ability of the proposed model to handle chunks as if they were one chunk comparing with the base model. As baseline we used T5 transformer. We studied the rule of memory slots augmented to each input chunk and studied the model performance without selector. We found that adding memory to input chunks helped the proposed model to overcome the baseline on Masked language modeling task with specific training parameters. Ablation study reveals the ability of using the compressed input chunks with a degradation in performance.