MCRM: Mother Compact Recurrent Memory

TL;DR

This paper introduces MCRM, a nested LSTM-GRU architecture that enhances memory capabilities by combining long-term and short-term neuron functions, showing improved performance on specific tasks.

Contribution

The paper proposes MCRM, a novel nested LSTM-GRU architecture with a compact memory pattern, integrating forget and input gates as input to the GRU, advancing recurrent neural network design.

Findings

MCRM outperforms previous architectures on certain tasks.

MCRM has a compact memory pattern with neurons functioning in both long-term and short-term modes.

Empirical results demonstrate the effectiveness of MCRM in sequence modeling.

Abstract

LSTMs and GRUs are the most common recurrent neural network architectures used to solve temporal sequence problems. The two architectures have differing data flows dealing with a common component called the cell state (also referred to as the memory). We attempt to enhance the memory by presenting a modification that we call the Mother Compact Recurrent Memory (MCRM). MCRMs are a type of a nested LSTM-GRU architecture where the cell state is the GRU hidden state. The concatenation of the forget gate and input gate interactions from the LSTM are considered an input to the GRU cell. Because MCRMs has this type of nesting, MCRMs have a compact memory pattern consisting of neurons that acts explicitly in both long-term and short-term fashions. For some specific tasks, empirical results show that MCRMs outperform previously used architectures.