
TL;DR
This paper proposes a comprehensive taxonomy for neural memory networks based on their memory organization, analyzing their expressive power and guiding their application to various tasks.
Contribution
It introduces a unified taxonomy for memory networks, compares their expressive capabilities, and connects these to task requirements for better network selection.
Findings
The taxonomy ranks networks by expressive power: RNN <= LSTM <= neural stack <= neural RAM.
Synthetic tasks validate the expressive power hierarchy.
Natural language processing examples illustrate taxonomy's practical utility.
Abstract
In this paper, a taxonomy for memory networks is proposed based on their memory organization. The taxonomy includes all the popular memory networks: vanilla recurrent neural network (RNN), long short term memory (LSTM ), neural stack and neural Turing machine and their variants. The taxonomy puts all these networks under a single umbrella and shows their relative expressive power , i.e. vanilla RNN <=LSTM<=neural stack<=neural RAM. The differences and commonality between these networks are analyzed. These differences are also connected to the requirements of different tasks which can give the user instructions of how to choose or design an appropriate memory network for a specific task. As a conceptual simplified class of problems, four tasks of synthetic symbol sequences: counting, counting with interference, reversing and repeat counting are developed and tested to verify our…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsSoftmax · Sigmoid Activation · Tanh Activation · Neural Turing Machine · Memory Network · Location-based Attention · Content-based Attention · Long Short-Term Memory
