Assessing the Memory Ability of Recurrent Neural Networks

TL;DR

This paper investigates the memory capabilities of various RNN units, introducing a Semantic Euclidean Space and evaluation metrics to analyze and compare their ability to retain semantic information over sequences.

Contribution

It proposes a novel Semantic Euclidean Space and evaluation indicators to assess and compare the memory abilities of different recurrent units.

Findings

Semantic Euclidean Space effectively represents sequence semantics.

Evaluation indicators reveal differences in memory capabilities among RNN units.

Guidelines for selecting sequence lengths during training are provided.

Abstract

It is known that Recurrent Neural Networks (RNNs) can remember, in their hidden layers, part of the semantic information expressed by a sequence (e.g., a sentence) that is being processed. Different types of recurrent units have been designed to enable RNNs to remember information over longer time spans. However, the memory abilities of different recurrent units are still theoretically and empirically unclear, thus limiting the development of more effective and explainable RNNs. To tackle the problem, in this paper, we identify and analyze the internal and external factors that affect the memory ability of RNNs, and propose a Semantic Euclidean Space to represent the semantics expressed by a sequence. Based on the Semantic Euclidean Space, a series of evaluation indicators are defined to measure the memory abilities of different recurrent units and analyze their limitations. These evaluation indicators also provide a useful guidance to select suitable sequence lengths for different RNNs during training.