A Novel Framework for Recurrent Neural Networks with Enhancing Information Processing and Transmission between Units

TL;DR

This paper introduces a biologically inspired RNN framework with three stages—working memory, forgetting, and long-term storage—that improves performance across various tasks by enhancing information processing and selective forgetting.

Contribution

The paper presents a new RNN framework inspired by human memory models, incorporating three stages to better manage information flow and improve task performance.

Findings

Improves performance on text, image, and language modeling tasks.

Effectively forgets secondary information during processing.

Enhances information transmission between RNN units.

Abstract

This paper proposes a novel framework for recurrent neural networks (RNNs) inspired by the human memory models in the field of cognitive neuroscience to enhance information processing and transmission between adjacent RNNs' units. The proposed framework for RNNs consists of three stages that is working memory, forget, and long-term store. The first stage includes taking input data into sensory memory and transferring it to working memory for preliminary treatment. And the second stage mainly focuses on proactively forgetting the secondary information rather than the primary in the working memory. And finally, we get the long-term store normally using some kind of RNN's unit. Our framework, which is generalized and simple, is evaluated on 6 datasets which fall into 3 different tasks, corresponding to text classification, image classification and language modelling. Experiments reveal that our framework can obviously improve the performance of traditional recurrent neural networks. And exploratory task shows the ability of our framework of correctly forgetting the secondary information.