Separation of Memory and Processing in Dual Recurrent Neural Networks

TL;DR

This paper introduces a dual recurrent neural network architecture that separates memory and processing, leading to simpler, more interpretable models with higher accuracy on various language and arithmetic tasks.

Contribution

It proposes a novel neural network design that stacks recurrent and feedforward layers, demonstrating improved interpretability and accuracy over traditional architectures.

Findings

Models behave as finite automata with binary activation under noise.

Achieve higher accuracy on language recognition, addition, and expression generation tasks.

Simpler models are easier to interpret and analyze.

Abstract

We explore a neural network architecture that stacks a recurrent layer and a feedforward layer that is also connected to the input, and compare it to standard Elman and LSTM architectures in terms of accuracy and interpretability. When noise is introduced into the activation function of the recurrent units, these neurons are forced into a binary activation regime that makes the networks behave much as finite automata. The resulting models are simpler, easier to interpret and get higher accuracy on different sample problems, including the recognition of regular languages, the computation of additions in different bases and the generation of arithmetic expressions.