Iterative evaluation of LSTM cells

TL;DR

This paper introduces an iterative modification to LSTM cells that enhances their performance by repeating computations over fixed inputs and states, improving language modeling capabilities efficiently.

Contribution

The paper proposes a novel iterative scheme for LSTM cells, significantly boosting performance without increasing parameter count substantially.

Findings

Improved language modeling performance

Comparable results with more than three times the parameters

Theoretical and empirical validation of the iterative approach

Abstract

In this work we present a modification in the conventional flow of information through a LSTM network, which we consider well suited for RNNs in general. The modification leads to a iterative scheme where the computations performed by the LSTM cell are repeated over a constant input and cell state values, while updating the hidden state a finite number of times. We provide theoretical and empirical evidence to support the augmented capabilities of the iterative scheme and show examples related to language modeling. The modification yields an enhancement in the model performance comparable with the original model augmented more than 3 times in terms of the total amount of parameters.