Understanding Memory Modules on Learning Simple Algorithms

TL;DR

This paper investigates how memory modules in neural networks enable learning simple algorithms, using visualization and dimension reduction to analyze strategies in neural Turing and stack-augmented networks.

Contribution

It introduces a novel qualitative analysis method to understand memory strategies and compares two memory-augmented neural networks on algorithm tasks.

Findings

Both models generalize on sequence reversal.

Only stack-augmented network generalizes on expression evaluation.

Different input categories trigger distinct memory policies.

Abstract

Recent work has shown that memory modules are crucial for the generalization ability of neural networks on learning simple algorithms. However, we still have little understanding of the working mechanism of memory modules. To alleviate this problem, we apply a two-step analysis pipeline consisting of first inferring hypothesis about what strategy the model has learned according to visualization and then verify it by a novel proposed qualitative analysis method based on dimension reduction. Using this method, we have analyzed two popular memory-augmented neural networks, neural Turing machine and stack-augmented neural network on two simple algorithm tasks including reversing a random sequence and evaluation of arithmetic expressions. Results have shown that on the former task both models can learn to generalize and on the latter task only the stack-augmented model can do so. We show that different strategies are learned by the models, in which specific categories of input are monitored and different policies are made based on that to change the memory.

Tables

Table 1. Table 1: Task examples. The 𝐱 i subscript 𝐱 𝑖 \mathbf{x}_{i} represents the i 𝑖 i th random binary vector. #LPO is short for number of low-priority operators. Note that for M10AE the / represents a modulo operator and each intermediate result is followed by a modulo- 10 10 10 operation. For an instance, the evaluation procedure of the example in the table can be done in 4 4 4 steps: i. ( 6 ∗ 3 ) % ​ 10 = 8 percent 6 3 10 8 (6*3)\%10=8 , ii. ( 8 % ​ 2 ) % ​ 10 = 0 percent percent 8 2 10 0 (8~{}\%~{}2)\%10=0 , iii. ( 8 + 0 ) % ​ 10 = 8 percent 8 0 10 8 (8+0)\%10=8 , iv. ( 8 − 4 ) % ​ 10 = 4 percent 8 4 10 4 (8-4)\%10=4 .

task	input	output	measure
mirror	${𝐱}_1{𝐱}_2\cdots {𝐱}_L$	${𝐱}_L{𝐱}_{L-1}\cdots {𝐱}_1$	input length
M10AE	$8+6\ast 3/2-4$	$4$	#LPO

Equations

$${\mathbf{i}}_t=f_i({\mathbf{x}}_t,{\mathbf{r}}_{t-1}),$$

$${\mathbf{h}}_t=\mathrm{LSTM}({\mathbf{i}}_t,{\mathbf{h}}_{t-1}).$$

$${\mathbf{o}}_t=f_o({\mathbf{o}}_t^{(c)},{\mathbf{r}}_t),$$

$${\mathbf{r}}_t={\mathbf{M}}_t[0].$$

$$\mathbf{M}:=\sum _{i}p(a_i|\mathbf{M})\cdot {\mathbf{M}}_{a_i},$$

$$p(\mathbf{a}|{\mathbf{M}}_t,{\mathbf{x}}_t)=\mathrm{softmax}({\mathbf{W}}_{a\mathbf{M}}Conv1d({\mathbf{M}}_t)+{\mathbf{W}}_{ax}{\mathbf{x}}_t+{\mathbf{b}}_a),$$

$${\mathbf{r}}_t=\sum _{i=0}^{N-1}{\mathbf{w}}_t^r[i]\mathbf{M}[i],$$

$${\mathbf{M}}_t={\mathbf{M}}_{t-1}\odot (1-{\mathbf{w}}_t^w{\mathbf{e}}_t^T)+{\mathbf{w}}_t^w{\mathbf{a}}_t^T,$$

$$p_t=\sum _{i=0}^{N-1}{\mathbf{w}}_t[i]\cdot i,$$

$$P(\mathrm{PUSH}|{\mathbf{M}}_t)=\sum _{k=0}^{K-1}P({\mathrm{PUSH}}_k|{\mathbf{M}}_t),$$

$$n_{\mathrm{pop}}=\sum _{k=0}^{K-1}P({\mathrm{POP}}_k|{\mathbf{M}}_t)\cdot k.$$

Full text

1 Introduction

The generalization ability of neural networks is the most important criterion to determine whether they are powerful or not. Recent work on memory-augmented neural networks (MANN) has shown promising results about generalizing perfectly on simple-algorithm tasks (Grefenstette et al., 2015; Joulin and Mikolov, 2015; Graves et al., 2016; Rae et al., 2016; Gulcehre et al., 2017). However, understanding the working mechanism of memory modules has not been well studied. Existing work has stopped at conjecturing the underlying learned strategies based on shallow, single-case visualization without any further verification. This raises concern of the lack of interpretability and hinders us from designing better memory modules. Currently, there are two main difficulties for interpreting the black-box memory-modules.

Firstly, the diversity among the different MANNs makes it hard to isolate the functions of memory-modules. Consequently, we cannot focus on the decisive parts of the models, as the different designs make the comparison work hard to carry out. Secondly, to interpret MANNs is challenging. Although interpretation methods for RNN models are well studied, little attention has been paid on that for MANNs.

To solve the above problems, we formalize a unified framework for MANNs with different implementations of memory modules, which makes comparison among different memory modules feasible. Then, we propose a novel qualitative analysis method based on dimension reduction for interpreting memory cells by verifying hypotheses. We implement neural Turing machine and stack-augmented neural network under the unified framework and carry out detailed analysis on two algorithm tasks consisting of reversing a random sequence and evaluating arithmetic expressions to show the effectiveness of our proposed analysis method.

The experiment and analysis have shown that the external memory can compensate for the need for storing intermediate results to travel along the dependency path. Specifically, neural Turing machine generalizes well on mirror task, and stack-augmented neural network generalizes well on reversing a random sequence and evaluation of arithmetic expressions. To summarize, our main contributions are as follows:

• •

We generalize different MANNs with a unified framework, which ensures the fairness of comparing different memory mechanisms.

• •

We propose a novel analysis method for memory cells. By applying our analysis method, we infer and verify the hypotheses about what strategy has been learned by the model that can generalize well.

2 Our unified MANN framework

In order to compare different types of memory modules, we propose a unified framework for MANNs by fixing the controller and memory access interface. By abstracting the processing components used in stack augmented neural network (Joulin and Mikolov, 2015; Yogatama et al., 2018) and neural Turing machines (Graves et al., 2014), the framework shown in Figure 1 contains an LSTM controller, a memory module at time $t$ with $N$ cells represented as $\mathbf{M}_{t}=[\mathbf{m}_{0}\ \mathbf{m}_{1}\ \cdots\ \mathbf{m}_{N-1}]^{T}\in\mathbb{R}^{N\times d}$ equipped with a specific read-write method, such as push and pop actions of the stack memory extension.

The input $\mathbf{i}_{t}$ at time step $t$ to the controller is a combination of the system input $\mathbf{x}_{t}$ at time step $t$ and readout $\mathbf{r}_{t-1}$ at the last time step. Formally,

[TABLE]

where $f_{i}$ is a learnable function and we here take it simply to be a concatenation operation. The state of the LSTM controller is represented as $\mathbf{h}_{t}$, updated by the standard LSTM model:

[TABLE]

The controller output $\mathbf{o}_{t}^{(c)}=\mathbf{h}_{t}$ and the input $\mathbf{x}_{t}$ are then inputted to the write module and the read module, indicating how to interact with the memory. Finally, the readout $\mathbf{r}_{t}$ is combined with the controller output $\mathbf{o}_{t}^{(c)}$ to form the system output $\mathbf{o}_{t}$ at time step $t$:

[TABLE]

where $f_{o}$ is also a concatenation function.

In this paper, we experiment with two typical MANNs whose external memories are a stack memory and a tape memory respectively. To implement the two models under our proposed framework, we only need to specify the detailed read and write methods, which are shown in Section 2.1 and 2.2 respectively. As a special case, an LSTM model is an instance of the framework without memory111The LSTM model contains internal memory actually. The term memory mentioned in this paper all refers to external memory, if without specific explanation..

2.1 Stack memory

We adopt the stack-augmented neural network (SANN) proposed in Yogatama et al. (2018). The readout at each time step is just the top of the stack:

[TABLE]

For each write step, there are $2K+2$ possible actions $a_{i}$’s to choose:

• •

$\mathrm{PUSH_{k}}$: Push the transformed current controller state $f_{s}(\mathbf{h}_{t})$ onto the stack after $k\ (k=0,1,2,\cdots K)$ pops.

• •

$\mathrm{STAY_{k}}$: Keep the stack unchanged after $k\ (k=0,1,2,\cdots K)$ pops.

Where $f_{s}$ is a learnable function and is implemented as a linear transformation.

In order to ensure the model differentiable, the write step for the stack is formalized as the expectation of the memory after one of the $2K+2$ specific actions $a_{i}$:

[TABLE]

where the $\mathbf{M}_{a_{i}}$ is the memory after the action $a_{i}$ is adopted, and the probabilities of the write actions $\mathbf{a}=[a_{0}\ a_{1}\ \cdots\ a_{2K+1}]^{T}$ is computed using the current memory $\mathbf{M}_{t}$ and the system input $\mathbf{x}_{t}$222Although the policy is parameterized in a recursive formula in the original paper, we find this simple setting is powerful enough.:

[TABLE]

where the $\mathrm{Conv1d}$ is a two-channel $1$-D convolutional operation with a size-$2$ kernel over the $N$ memory cells.

2.2 Tape memory

As the memory in a standard Turing machine is called a tape, we here name the class of neural Turing machines by TANNs, i.e. tape augmented neural networks. The tape memory module is adopted as mentioned in Graves et al. (2014), with random access read-write steps totally controlled by the controller.

Both the read and write actions contain two preparation steps before the actual interaction with the memory: 1) an analysis step to the controller state and 2) an addressing step. The detailed formulas of these two steps are omitted here. After the addresses to read and write are determined, the readout is the expectation of the memory cells over distribution $\mathbf{w}^{r}$:

[TABLE]

where $[i]$ means to index the $i$th row of the vector or matrix. And the write step:

[TABLE]

which is combining the influence of erase vector $\mathbf{e}_{t}$ and add vector $\mathbf{a}_{t}$ on the memory $\mathbf{M}_{t-1}$ over the distribution $\mathbf{w}^{w}$.

3 Experiment

In the experiment, we want to figure out which kind of and how memory module helps generalize on two algorithm tasks including reversing a random sequence (called mirror task) and evaluating arithmetic expressions (called M10AE task). Simple RNN (SimpRNN) and LSTM are adopted as two baseline models, which represent neural networks without external memory modules.

3.1 Experimental settings

For mirror task, the input is a sequence of binary vectors whose size is $9$. During encoding (input) stage, the inputs are randomly sampled from Bernoulli distribution with $p=0.5$ and the inputs are zero vector during decoding (output) stage. This setting is similar to that of copy task in Graves et al. (2014). Both the number $N$ of and size $M$ of memory cells are $20$. The controller dimension is $100$. For M10AE task, the input embeddings $\mathbf{x}_{t}$’s are trainable parameters from random initialization. The input embedding dimension $100$. The dimension of a memory cell and a controller state is set to be equal to the input embedding dimension. The number of memory cells is chosen from [$5$, $10$, $20$]. We adopt training using Adam algorithm with batch size chosen from [$32$, $64$, $128$, $256$] and the learning rate chosen from [$0.001$, $0.0001$]. The hyper-parameters are tuned on a development set.

3.2 Mirror

First, we are interested in exploring whether SANN and TANN are able to learn to output the input sequence in a reverse order, which we call mirror task. An example of this task is shown in the first row of Table 1. We append a delimiter at the end of the input to tell the model when to output, and it is noted as $\langle\mathrm{EOS}\rangle$. We adopt the length of input sequence as the difficulty measure of each sample.

3.2.1 Result

The maximum length of input sequence is $5$ during the training stage, and the maximum length is extended to $10$ when testing. Here we view a prediction as correct only if the whole output sequence is the same as that of the input. The result is shown in Figure 2. We can find that both SANN and TANN can generalize beyond input length of training samples. Figure 2b shows that SANN converges faster than TANN, which corresponds with the intuition that stack memory is more suitable for this task.

3.2.2 Analysis

Since TANN and SANN can generalize greatly, we here analyze both of them to investigate what strategy they have induced. In order to gain a general averaged insight into what mechanism underlying these two models on mirror task, we generate $500$ samples with the same length, whose each input binary vector is restricted in the binary format of $1,2,\cdots,9$. These numbers can be viewed as the labels of the samples, which helps index the input vectors. And all the analysis for mirror task is based on the $500$ samples.

We first are interested in investigating how the controller gates change on mirror task. Specifically, we plot in Figure 3a and 3c the averaged saturation ratio (Karpathy et al., 2015) of the input gates and forget gates of the controller along with each input. Here a gate is defined right-saturated if its value is larger than $0.9$ and defined left-saturated if its value is less than $0.1$. Comparing these two figures, we can find that both TANN and SANN are sensitive of the delimiter in terms of each controller gates, after which dramatic changes of saturation rate appear. The change of controller gates of TANN seem more complicated than that of SANN, which indicates it is much easier to control a stack memory to finish the mirror task than a tape memory. The early convergence of SANN on mirror task can also support this idea.

We then visualize the read-write and push-pop policies for TANN and SANN respectively. For read-write policy of TANN, we average the expected address over the $500$ samples. The expected address $p_{t}$ for read and write operations at time step $t$ can be calculated as:

[TABLE]

where the $\mathbf{w}_{t}$ is either $\mathbf{w}_{t}^{r}$ or $\mathbf{w}_{t}^{w}$ to get the expected address for read or write. For SANN, the push probability is just the probability sum mass of all type of push actions:

[TABLE]

and the expected number of times to pop (noted as $n_{\mathrm{pop}}$) at time step $t$ can be calculated as:

[TABLE]

The result is shown in Figure 3b and 3d. We can find that both for TANN and SANN the policies for encoding (solid red lines) and decoding (solid blue lines) information are opposite to each other, indicating reversing in their own ways.

Based on these findings above, we make hypothesis about what strategy learned by TANN by the following pseudocode:

1:sequence of binary vectors $s=\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{L}$

2:the reverse of the input sequence

3:$\mathbf{M}\in\mathbb{R}^{N\times M}$ $\triangleright$ initialize the memory randomly

4:$w^{w}\leftarrow 17$ $\triangleright$ initialize the writing-to address

5:for  each input $\mathbf{x}_{t}$ in $s$ do $\triangleright$ encoding stage

6:     if $\mathbf{x}_{t}$ is a delimiter then $\triangleright$ the end of the input

7:         $w^{r}\leftarrow(17+L-1)\%N$ $\triangleright$ initialize the reading-from address

8:     else

9:         $\mathbf{M}[w^{w}]\leftarrow\mathbf{x}_{t}$ $\triangleright$ write to store $\mathbf{x}_{t}$

10:         $w^{w}\leftarrow(w^{w}+1)\%N$ $\triangleright$ point to the next cell

11:     end if

12:end for

13:for $t$ in $1,2,...,L$ do $\triangleright$ decoding stage

14:     output $\mathbf{M}[w^{r}]$ $\triangleright$ read and output

15:     $w^{r}\leftarrow(w^{r}-1)\%N$ $\triangleright$ point to the next cell

16:end for

And the hypothesis strategy for SANN on mirror task is:

1:sequence of binary vectors $s=\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{L}$

2:the reverse of the input sequence

3:empty the stack $S$

4:for  each input $\mathbf{x}_{t}$ in $s$ do $\triangleright$ encoding stage

5:     if $\mathbf{x}_{t}$ is not a delimiter then

6:         push the $\mathbf{x}_{t}$ to $S$ $\triangleright$ store information

7:     end if

8:end for

9:for $t$ in $1,2,...,L$ do $\triangleright$ decoding stage

10:     output the top of $S$ $\triangleright$ read and output

11:     pop the top of $S$

12:end for

The next goal is to verify these hypotheses above. To this end, we evaluate the hypothesis information encoded in each memory cell by our proposed qualitative verification method. This method is based on the assumption that if the intermediate results are the same in a certain step of the hypothesized strategy, then their distributed representations in the memory should be similar as well. In detail, this includes $4$ steps:

Collecting the memory cells $[\mathbf{m}_{t}^{(1)},\mathbf{m}_{t}^{(2)},\cdots,\mathbf{m}_{t}^{(l)}]$ at the same position in the memory at the same time step. 2. 2.

Labelling the memory cells with the corresponding results $[\hat{y}_{t}^{(1)},\hat{y}_{t}^{(2)},\cdots,\hat{y}_{t}^{(l)}]$ derived from the candidate strategy to get pairs $\left\{(\mathbf{m}_{t}^{(i)},\hat{y}_{t}^{(i)})\right\}_{i=1}^{l}$. 3. 3.

Using t-SNE (Maaten and Hinton, 2008) to visualize the labelled memory cell vectors. 4. 4.

If there appear the clear labelled clusters, then the candidate strategy is reasonable.

The examples are shown in Figure 4. The compact labelled clusters in Figure 4a, 4b (for TANN) and Figure 4d, 4e (for SANN) support the hypothesis semantics shown in the captions of each subfigure. We also include negative examples in Figure 4c and 4f to exemplify when the evaluation result shows the hypothesis is not reasonable. Since the input sequence are randomly sampled, labelling guided by a wrong hypothesis can not cover all the samples and this mismatch will also show up in the visualization as chaotic labels.

3.3 M10AE

After analysis on mirror tasks, we are then interested in extending the interpretation procedure to the harder case. We propose a new simple algorithm task named modulo-$10$ arithmetic expressions, M10AE, in which each input sequence is in the form of an arithmetic expression without parentheses and the output is the evaluation result of it. In order to make the task tractable to analyze while preserving the recursive nature inside it, we add the following constraints:

• •

Each numeral is limited in $1,2,\cdots,9$.

• •

The / represent a modulo operator.

• •

Each intermediate result is modulo by $10$.

• •

The ***, / have higher priority than + and -.

An example is shown in the second row of Table 1. Based on the constraints, the evaluation results can only range in integers from [math] to $9$, and thus we formalize the task as a $10$-class classification. Since the computation procedure interrupts when encountering low-priority operators and thus larger number of low-priority operators (#LPO) means larger memory burden, we take #LPO to be the difficulty measure for M10AE.

Recently, Hupkes et al. (2018) and Jacob et al. (2018) have proposed two similar tasks. However, they formalize the task as either a regression problem or a two-digit sequence prediction problem, without consideration of the intermediate results. These settings are poor at restricting the space of intermediate results, whose distribution is much sparser, and this hinders us from verifying a candidate strategy empirically. By contrast, in M10AE, the result is given as a classification label, and both the intermediate results and the final results range in integers from [math] to $9$ due to modulo-$10$ design. This leads to abundant samples for each intermediate category.

3.3.1 Result

$80$ and $20$ thousand examples are generated for training and valiation, respectively. The maximum #LPO is $14$ for training and $20$ for validation. The results are shown in Figure 5. Overall, the performance of all the models except SANN drops quickly with the increase of #LPO. Figure 5a indicates the SANN has learned to generalize on this task. In addition, the models with an external memory are better than the ones without any external memory (i.e. LSTM and SimpRNN). By contrast, the baseline model simpRNN performs extremely bad, indicating that the internal memory of LSTM is crutial for this task. As shown in Figure 5b, all the models converge slowly, which indicates the complexity of the task.

3.3.2 Analysis

As the SANN seems to induce a stable strategy, we then analyze it. We generate $500$ samples with the same structure to get general averaged patterns. And the analysis is all based on these $500$ examples. These examples are noted as $\langle N_{0}\rangle\langle L_{0}\rangle\langle{N_{1}}\rangle\langle{H_{1}}\rangle\langle{N_{2}}\rangle\langle{H_{2}}\rangle\langle{N_{3}}\rangle\langle{L_{1}}\rangle\langle{N_{4}}\rangle$, where $\langle{N_{i}}\rangle$ represents the $i$th numeral, $\langle{L_{i}}\rangle$ represents the $i$th low-priority operator (i.e. + or -), and $\langle{H_{i}}\rangle$ represents the $i$th high-priority operator (i.e. *** or /). One samples is 8+63/2-4*. Other samples generated by substituting the operators and numbers at the same position into the other symbols from the same category, e.g., *** to /, + to -, 1 to 2, and etc..

In the same way with Section 3.2.2, we first visualize the controller gates and the read-write policy of SANN. The results are shown in Figure 6. As shown in Figure 6a, The controller gates peak at the time steps when the low-priority operators (i.e. $\langle L_{0}\rangle$ and $\langle L_{1}\rangle$ in Figure 6a) appear, which indicates the controller of SANN track specific categories of symbols as the controllers do in mirror task. As shown in Figure 6, the push/pop actions are adopted regularly with respect to local structures: 1) every time the lower-priority operator (+, -) comes, the push probability goes to zero and the expectation of pop times rise up sharply to around $3$; and 2) when dealing with high-priority operations (***, /), the push/pop lines go up/down with each numeral input relatively more gently.

However it is still hard to make a hypothesis about what SANN has learned, and we then visualize the averaged memory at each time step to get further clues shown in Figure 7. The pattern is highly regular: the stack pops all its stored items when it comes to the end of a term (i.e. $\langle{N_{0}}\rangle$ and $\langle{N_{1}}\rangle\langle{H_{0}}\rangle\langle{N_{2}}\rangle\langle{H_{1}}\rangle\langle{N_{3}}\rangle$ here), which is consistent with Figure 6b; during processing each term, the stack pushes every time it sees a high-priority operator.

A hypothesis about what strategy SANN has induced is shown below:

1:arithmetic expression $e=x_{0},x_{1},\cdots,x_{L}$

2:the evaluation result of $e$

3:$S\leftarrow[0,0,\cdots,0]^{T}\in\mathbb{R}^{N}$ $\triangleright$ stack

4:$l_{0}$, $l_{1}$ $\leftarrow$ [math] $\triangleright$ low/high-priority results

5:$l_{0}^{\prime}\leftarrow 0$ $\triangleright$ temporary low-priority result

6:$p_{0}$, $p_{1}$ $\leftarrow$ null $\triangleright$ low/high-priority operators

7:for  each input $x_{t}$ in $e$ do

8:     if $x_{t}\in\{+,-\}$ then $\triangleright$ for low-priority operators

9:         $p_{0}\leftarrow x_{t}$ $\triangleright$ save the operator

10:         pop till empty

11:     else if $x_{t}\in\{*,/\}$ then $\triangleright$ for high-priority operators

12:         $l_{0}\leftarrow l_{0}^{\prime}$ $\triangleright$ adopt the candidate result

13:         push ($l_{1}$, $x_{t}$) $\triangleright$ push the combination

14:     **else$\triangleright$ for numerals **

15:         $l_{1},p_{1}\leftarrow S[0]$ $\triangleright$ read the combination pushed

16:         $l_{1}\leftarrow\mathrm{eval}(p_{1},l_{1},x_{t})$ $\triangleright$ high-priority operation

17:         $l_{0}^{\prime}\leftarrow l_{0}$ $\triangleright$ save the candidate result

18:         $l_{0}\leftarrow\mathrm{eval}(p_{0},l_{0},l_{1})$ $\triangleright$ low-priority operation

19:         pop till empty

20:         push $l_{0}$ $\triangleright$ push the whole result so far

21:     end if

22:end for

23:return $l_{0}$

where $\mathrm{eval}(f,a_{0},a_{1})$333As a special case, if the stack is empty or $f$ is $\mathrm{null}$, the second argument of this function will be returned. is to evaluate the result of $f(a_{0},a_{1})$ given the function pointer $f$ and the arguments $a_{0}$, $a_{1}$.

There are two kinds of storage in this hypothesis, in-controller storage $l_{0},l_{0}^{\prime},l_{1},p_{0},p_{1}$ and in-memory storage $S$. An example is shown in Figure 8, where the boxes and arrows in purple (an evaluation step, e.g. 63%10=8* at time step $4$) and red (a combination-pushing step, e.g. push the combination of 6 and *** at time step $3$) directing the information flow indicates a recursive strategy.

As in Section 3.2.2, we then verified the hypothesis by our proposed analysis method. The result is shown Figure 9, where the clear clusters (in Figure 9a and 9b) indicate the hypothesis is reasonable. A negative example is also included in Figure 9c for illustration about an unreasonable hyothesis.

4 Related Work

The related work can be divided into two categories, memory-augmented neural networks and visualization methods.

4.1 Memory Augmented Neural Network

The first MANN model is NTM, neural Turing machine (Graves et al., 2014), which is proposed to assign logical flow control over external memory to RNNs. These types of models are associated with the automaton theory. The MANNs can be viewed as RNNs with only internal memory augmented with different types of data structures, as a simple DFA is to form complex automaton like PDA, LBA and Turing machine. Thus some work focus on various memory types from different prior bias for specific tasks. For example, an RNN can learn to generalize on context-free grammars augmented with stacks (Joulin and Mikolov, 2015; Grefenstette et al., 2015), learn to model shortest syntactic dependence with a gated memory (Gulcehre et al., 2017) and learn to solve shortest-path tasks with a more general tape memory (Graves et al., 2016). Some work is dedicated to overcoming the defects of these models, especially for NTMs, such as separating each memory cell into content and address vectors (Gulcehre et al., 2016), introducing memory allocation and de-allocation protocols (Graves et al., 2016; Munkhdalai and Yu, 2017), speeding up the addressing mechanism (Rae et al., 2016) and adding adaptive computational design (Yogatama et al., 2018).

4.2 Understanding Recurrent Networks

Understanding the RNNs and their variants is enjoying renewed interest, as a result of successful applications in a wide range of machine learning problems on sequential data. For one thing, many work focus on what the RNNs remember in their hidden states. Some visualize the dynamics of the LSTM gates in terms of absolute value and saturation rates that keep track of the structure hints (Karpathy et al., 2015; Ghaeini et al., 2018). And some plot t-SNE visualization for clause representations and derivative saliency maps to understand the polarity changes in sentiment analysis (Li et al., 2016a). There are also researches trying training multiple decoders to predict the history inputs for checking what and how much information is stored (Koppula et al., 2018). And in Verwimp et al. (2018), the gradient matrix of the states with respect to the input embeddings is decomposed with SVD to find the principal direction in the input space. For another, which part of the input has a key effect on the model decisions is also a popular topic. Many works view this problem as searching the minimum set of word vectors or their dimensions to flip the models’ decision (Li et al., 2016b; Westhuizen and Lasenby, 2018); and there is other work computing different relevance measures between inputs and outputs to explore the contribution of the words (Arras et al., 2017; Ding et al., 2017; van der Westhuizen and Lasenby, 2017).

5 Conclusion

In this paper, we analyze strategies learned by memory augmented neural networks on task of reversing random sequence and evaluation of arithmetic expressions. By visualizing the controller gates and read-write policy for the memory, we find both models can summarize the input symbols into categories and dynamically change the policy according to these categories. We make hypothesis about what strategy is induced by the models, and verifying them by our proposed novel qualitative analysis method. One can mimic the analysis pipeline for other settings and thus this work helps inspire more researches on the strategy interpretation for MANNs.