Network of Recurrent Neural Networks

TL;DR

This paper introduces Network Of Recurrent neural networks (NOR), a new hierarchical RNN architecture inspired by systems theory, demonstrating superior performance over simple RNNs and competitive results with GRU and LSTM.

Contribution

The paper proposes a novel hierarchical RNN architecture called NOR, based on systems theory, with new design methodologies for different topologies.

Findings

NOR outperforms simple RNNs with the same parameters

NOR sometimes surpasses GRU and LSTM in performance

Experiments on three tasks validate the effectiveness of NOR

Abstract

We describe a class of systems theory based neural networks called "Network Of Recurrent neural networks" (NOR), which introduces a new structure level to RNN related models. In NOR, RNNs are viewed as the high-level neurons and are used to build the high-level layers. More specifically, we propose several methodologies to design different NOR topologies according to the theory of system evolution. Then we carry experiments on three different tasks to evaluate our implementations. Experimental results show our models outperform simple RNN remarkably under the same number of parameters, and sometimes achieve even better results than GRU and LSTM.