TransfoRNN: Capturing the Sequential Information in Self-Attention Representations for Language Modeling

TL;DR

This paper introduces TransfoRNN, a model combining recurrent neural networks with Transformers to explicitly capture sequential information, resulting in improved language modeling performance with fewer parameters.

Contribution

The paper proposes a novel cascade of RNNs and Transformers, called TransfoRNN, which effectively captures sequential information and achieves comparable or better results with smaller models.

Findings

Lower perplexities on Penn Treebank and WikiText-2 datasets.

Reduced model size while maintaining performance.

Comparable results on LibriSpeech speech recognition task.

Abstract

In this paper, we describe the use of recurrent neural networks to capture sequential information from the self-attention representations to improve the Transformers. Although self-attention mechanism provides a means to exploit long context, the sequential information, i.e. the arrangement of tokens, is not explicitly captured. We propose to cascade the recurrent neural networks to the Transformers, which referred to as the TransfoRNN model, to capture the sequential information. We found that the TransfoRNN models which consists of only shallow Transformers stack is suffice to give comparable, if not better, performance than a deeper Transformer model. Evaluated on the Penn Treebank and WikiText-2 corpora, the proposed TransfoRNN model has shown lower model perplexities with fewer number of model parameters. On the Penn Treebank corpus, the model perplexities were reduced up to 5.5% with the model size reduced up to 10.5%. On the WikiText-2 corpus, the model perplexity was reduced up to 2.2% with a 27.7% smaller model. Also, the TransfoRNN model was applied on the LibriSpeech speech recognition task and has shown comparable results with the Transformer models.