A Taxonomy for Neural Memory Networks

TL;DR

This paper proposes a comprehensive taxonomy for neural memory networks based on their memory organization, analyzing their expressive power and guiding their application to various tasks.

Contribution

It introduces a unified taxonomy for memory networks, compares their expressive capabilities, and connects these to task requirements for better network selection.

Findings

The taxonomy ranks networks by expressive power: RNN <= LSTM <= neural stack <= neural RAM.

Synthetic tasks validate the expressive power hierarchy.

Natural language processing examples illustrate taxonomy's practical utility.

Abstract

In this paper, a taxonomy for memory networks is proposed based on their memory organization. The taxonomy includes all the popular memory networks: vanilla recurrent neural network (RNN), long short term memory (LSTM ), neural stack and neural Turing machine and their variants. The taxonomy puts all these networks under a single umbrella and shows their relative expressive power , i.e. vanilla RNN <=LSTM<=neural stack<=neural RAM. The differences and commonality between these networks are analyzed. These differences are also connected to the requirements of different tasks which can give the user instructions of how to choose or design an appropriate memory network for a specific task. As a conceptual simplified class of problems, four tasks of synthetic symbol sequences: counting, counting with interference, reversing and repeat counting are developed and tested to verify our arguments. And we use two natural language processing problems to discuss how this taxonomy helps choosing the appropriate neural memory networks for real world problem.