# Understanding Memory Modules on Learning Simple Algorithms

## TL;DR

This paper investigates how memory modules in neural networks enable learning simple algorithms, using visualization and dimension reduction to analyze strategies in neural Turing and stack-augmented networks.

## Contribution

It introduces a novel qualitative analysis method to understand memory strategies and compares two memory-augmented neural networks on algorithm tasks.

## Key findings

- Both models generalize on sequence reversal.
- Only stack-augmented network generalizes on expression evaluation.
- Different input categories trigger distinct memory policies.

## Abstract

Recent work has shown that memory modules are crucial for the generalization ability of neural networks on learning simple algorithms. However, we still have little understanding of the working mechanism of memory modules. To alleviate this problem, we apply a two-step analysis pipeline consisting of first inferring hypothesis about what strategy the model has learned according to visualization and then verify it by a novel proposed qualitative analysis method based on dimension reduction. Using this method, we have analyzed two popular memory-augmented neural networks, neural Turing machine and stack-augmented neural network on two simple algorithm tasks including reversing a random sequence and evaluation of arithmetic expressions. Results have shown that on the former task both models can learn to generalize and on the latter task only the stack-augmented model can do so. We show that different strategies are learned by the models, in which specific categories of input are monitored and different policies are made based on that to change the memory.