The unreasonable effectiveness of the forget gate

TL;DR

This paper demonstrates that a simplified LSTM variant with only a forget gate, called JANET, not only reduces computational complexity but also outperforms the standard LSTM on key benchmark datasets.

Contribution

The paper introduces JANET, a simplified LSTM model with only a forget gate, showing it can outperform standard LSTM in accuracy and efficiency.

Findings

JANET achieves 99% on MNIST and 92.5% on pMNIST.

JANET outperforms standard LSTM in benchmark tests.

Simplified gate structure improves performance and reduces computation.

Abstract

Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chrono-initialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models. Our proposed network, the JANET, achieves accuracies of 99% and 92.5% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5% and 91%.