Multiplicative LSTM for sequence modelling

TL;DR

The paper introduces multiplicative LSTM (mLSTM), a new recurrent neural network architecture that enhances sequence modelling by combining LSTM and multiplicative RNNs, demonstrating superior performance on language modelling tasks.

Contribution

It presents the novel mLSTM architecture that allows different recurrent transition functions for each input, improving expressiveness and performance in sequence modelling.

Findings

Outperforms standard LSTM on character-level language modelling tasks.

Achieves 1.27 bits/char on text8 dataset.

Attains a word perplexity of 88.8 on WikiText-2.

Abstract

We introduce multiplicative LSTM (mLSTM), a recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible input, which we argue makes it more expressive for autoregressive density estimation. We demonstrate empirically that mLSTM outperforms standard LSTM and its deep variants for a range of character level language modelling tasks. In this version of the paper, we regularise mLSTM to achieve 1.27 bits/char on text8 and 1.24 bits/char on Hutter Prize. We also apply a purely byte-level mLSTM on the WikiText-2 dataset to achieve a character level entropy of 1.26 bits/char, corresponding to a word level perplexity of 88.8, which is comparable to word level LSTMs regularised in similar ways on the same task.