Simple Recurrence Improves Masked Language Models

TL;DR

This paper introduces a simple recurrent module into Transformer models, demonstrating that recurrence enhances performance and stability across multiple tasks without increasing model complexity.

Contribution

The paper proposes a straightforward recurrent extension to Transformers, showing consistent performance gains and improved fine-tuning stability without additional parameters.

Findings

Recurrence improves Transformer performance by 2.1 points on average across 10 tasks.

Recurrent Transformer models show increased stability during fine-tuning.

The approach maintains the same number of parameters as standard Transformers.

Abstract

In this work, we explore whether modeling recurrence into the Transformer architecture can both be beneficial and efficient, by building an extremely simple recurrent module into the Transformer. We compare our model to baselines following the training and evaluation recipe of BERT. Our results confirm that recurrence can indeed improve Transformer models by a consistent margin, without requiring low-level performance optimizations, and while keeping the number of parameters constant. For example, our base model achieves an absolute improvement of 2.1 points averaged across 10 tasks and also demonstrates increased stability in fine-tuning over a range of learning rates.