Learning distant cause and effect using only local and immediate credit assignment

TL;DR

This paper introduces a biologically-plausible recurrent neural network with sparse coding that effectively learns distant cause-and-effect relationships, predicts sequences, and aids navigation with less memory than traditional models.

Contribution

The proposed network uses local, immediate credit assignment and sparse coding to encode sequences, enabling learning of long-range dependencies with reduced memory requirements.

Findings

Successfully associates distant causes and effects in stochastic processes.

Predicts higher-order sequences with partial observability.

Reduces memory usage compared to LSTM, GRU, and autoregressive models.

Abstract

We present a recurrent neural network memory that uses sparse coding to create a combinatoric encoding of sequential inputs. Using several examples, we show that the network can associate distant causes and effects in a discrete stochastic process, predict partially-observable higher-order sequences, and enable a DQN agent to navigate a maze by giving it memory. The network uses only biologically-plausible, local and immediate credit assignment. Memory requirements are typically one order of magnitude less than existing LSTM, GRU and autoregressive feed-forward sequence learning models. The most significant limitation of the memory is generalization to unseen input sequences. We explore this limitation by measuring next-word prediction perplexity on the Penn Treebank dataset.

Tables

Table 1. TABLE I: Distant cause & effect prediction accuracy on the Embedded Reber Grammar task for various neural models

Model	Accuracy
Elman RNN [10]	0
LSTM [10]	100
HTM [21]	98.4
RSM	99.2

Table 2. TABLE II: MNIST image sequence prediction experiment

Sequence	Markov order	Accuracy
‘0,1,2,3,4,5,6,7,8,9’	1	99.9
‘0,1,2,3,4, 0,4,3,2,1’	2	99.9
‘0,1,2,3, 0,1,2,3, 0,3,2,1’	5	99.9
None (image classification)	0	98.2

Table 3. TABLE III: Next-frame prediction error for the bouncing-balls task

Model	Error
Previous frame [30]	$11.82\pm 0.27$
RTRBM [27, 28]	$3.88\pm 0.33$
SRTRBM [28]	$3.31\pm 0.33$
DTSBN [29]	$2.79\pm 0.39$
PGN [30]	$0.65\pm 0.11$
LSTM [26]	$111.09\pm 0.68$
ConvLSTM [26]	$0.58\pm 0.16$
Seq2seq ConvLSTM [26]	$1.34\pm 0.19$
Seq2seq ConvLSTM multi-decoder [26]	$4.55\pm 0.40$
Conv RSM 1L	$3.51\pm 0.14$
Conv RSM 2L	$3.72\pm 0.29$
Conv RSM 1L + GAN	$0.8\pm 0.07$
Conv RSM 2L + GAN	$0.41\pm 0.05$

Table 4. TABLE IV: PTB test-set perplexity of RSM and later bRSM [ 24 ] models, with selected other algorithms for context

Model	Perplexity
Random clusterings LM [31]	170
RSM	166
5-gram Good-Turing [31]	162
Structured LM [31]	146
KN5 (our impl.)	143
Max. entropy 5-gram [31]	142
Feed-forward ANN LM (deep BP) [31]	140
Random forest LM [31]	131
RNN LM w. BPTT [31]	124
RSM + KN5	124
bRSM [24]	103
LSTM variants [33]	$58\le 86$
Transformer (GPT-2), varying size [34]	$35\le 65$

Equations

$${\mathbf{z}}^F=w^F{\mathbf{x}}^F(t)$$

$$z^R=w^R{\mathbf{x}}^R(t)$$

$$z^B=w^B{\mathbf{x}}^B(t)$$

$${\sigma }_{ij}={\mathbf{z}}_i^F+z_{ij}^R+z_{ij}^B$$

$${\pi }_{ij}=(1-{\varphi }_{ij}(t))\cdot ({\sigma }_{ij}-\min (\sigma )+1)$$

$${\pi }_i^G=\max ({\pi }_{i1},\dots ,{\pi }_{in})$$

$$M^C=\mathrm{topk}(\pi ,1)$$

$$M^G=\mathrm{topk}({\pi }^G,k)$$

$${\mathbf{y}}_{ij}=\tanh ({\sigma }_{ij}\cdot M_i^G\cdot M_{ij}^C)$$

$${\varphi }_{ij}(t+1)=\max ({\varphi }_{ij}(t)\cdot \gamma ,{\mathbf{y}}_{ij})$$

$$\psi (t)=\max (\psi (t-1)\cdot ϵ,\mathbf{y})$$

$${\mathbf{x}}^R(t+1)=\alpha \cdot \psi (t)$$

$${\mathbf{y}}_i^G=\max ({\mathbf{y}}_{i1},\dots ,{\mathbf{y}}_{in})$$

$${\hat{\mathbf{x}}}^F(t)=w^D{\mathbf{y}}^G$$

Full text

©2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

I Introduction

Researchers have been unable to find a biological equivalent to the deep error backpropagation used widely in artificial neural networks [1, 2]. This presents a credit assignment problem: How do biological neurons determine the influence of a synapse on an error that may occur many layers distant, or many steps in the future? The answer to this question is of practical significance because it may yield alternative, superior learning rules. Existing biologically-plausible approaches to the distant credit assignment problem have noticeable performance limitations when compared to implausible learning rules [3, 4, 5]. In this paper, we hope to overcome some of these limitations while preserving biological plausibility, using the approach of combinatoric sparse coding.

I-A Sequence learning

To date, the most successful architectures for sequence learning are Recurrent Neural Networks (RNNs) and auto-regressive, feed-forward networks [6]. Truncated Back-Propagation Through Time (BPTT) [7] unrolls the state of an RNN over a fixed number of time steps $t$, allowing errors to be assigned to the relevant synaptic weights. The network cannot be trained to exploit causes earlier than $t$ time-steps.

Autoregressive, feed-forward networks utilize a fixed moving window of $t$ most recent inputs, that are processed simultaneously through several layers. Error gradients are back-propagated through the layers. Like truncated BPTT, context more distant than $t$ steps cannot be utilized. Techniques such as dilated convolutions allow $t$ to be large (100 steps or more).

Credit assignment in the above methods is widely believed to be biologically implausible due to backpropagation through time and layers. Recent work based on Predictive Coding has shown that deep-backpropagation through network layers can be closely approximated using only local computations, but the approach cannot be extended to approximate BPTT [millidge2020predictive].

Credit assignment over time can also be achieved by forward-propagating derivatives of hidden states and outputs with respect to each synaptic weight. Real-Time Recurrent Learning (RTRL) [8] is an instance of this concept, but is rarely used in practice due to the difficulty of maintaining viable derivatives. A more recent example is e-prop (forward propagation of eligibility traces) but this is only shown to work for short-term association [9].

I-B Gated memory cells

In most RNNs, such as Elman networks, iterative partial derivative updates are likely to produce vanishing or exploding gradients. Gated memory layers such as Long-Short-Term-Memory (LSTM) [10] and Gated Recurrent Units (GRUs) [11] mitigate this problem by creating a more robustly differentiable relationship between memory state, gate operations and outputs.

Using gated memory cells, causes and effects may be successfully associated despite separation by hundreds of steps. However, the memory requirements of gated memory layers are much larger than BPTT with ordinary RNNs, due to the requirement to train many gate-control parameters.

Since we focus on the credit assignment problem, our objective is to demonstrate performance comparable or better than RNNs trained by BPTT. Gated memory architectures trained by BPTT likely represent an upper bound on our expectations, especially considering domain-specific optimizations that our model will lack.

I-C Practical requirements

How much context do we actually need for human-like sequence learning performance? Let’s take natural language modelling as an example. Chelba et al [12] showed that on the Billion Word benchmark, an LSTM $n$-gram model where $n=13$ was as good as LSTMs with access to longer history. In fact, even with simple smoothed $n$-gram language models such as Kneser-Ney [13], longer histories do not yield practical advantage, in part because the additional higher-order context is unlikely to generalize well, especially in smaller datasets.

Recent work by Bai et al [14] showed that in general, feed-forward autoregressive models with dilated convolutions tend to outperform RNNs. They also showed that even where very long term context was definitely necessary, RNNs could not exploit it. This implies that the encoding of context is at least as important as the amount of context.

We draw two conclusions. First, it is essential to verify model performance on real tasks to verify effective access to earlier context. Second, for competitive performance on natural language benchmarks it is likely only necessary to perform credit assignment over a relatively modest context of, perhaps, 100 steps. Since natural languages are tailored to human capabilities, this may be indicative of general human limits. Psychological studies of human problem-solving suggest that we work with relatively simple mental simulations of short sequences of highly abstracted features - longer sequences are made shorter by abstraction [15].

I-D Biologically plausible criteria

We adopted the following criteria for biological plausibility:

• •

Only local credit assignment. No back-propagation of errors between cell-layers

• •

No synaptic memory beyond the current and/or next step

• •

No time-travel, making use of past or future inputs or earlier hidden states

Note that we only aim to address the distant credit assignment problem, and do not claim that our approach is otherwise an accurate reproduction of biological network learning.

The computational capabilities of single-layer networks are very limited, especially in comparison to two-layer networks. Biological neurons perform “dendrite computation”, involving integration and nonlinearities within dendrite subtrees [16]. This is computationally equivalent to 2 or 3 typical artificial neural network layers. For this reason we allow ourselves to use error backpropagation across two ANN layers, under the assumption that this could approximate dendrite computation within a single biological cell layer, and training signals inside cells.

I-E Memorization approach

Backpropagation through time enables networks to selectively access hidden states and input from many steps prior. If we can only back-propagate errors across 2 or 3 time steps, how can we learn to retain data that won’t be useful for tens or hundreds of steps?

One option is to simply remember everything, which means we could use any past observation to predict a future effect. At first glance this seems impractical, even if we were to limit observations to one hundred steps; but Hawkins & Ahmad [17] showed that a form of sparse coding can allow networks to have vast representational capacity, by virtue of all the combinations in which cells can be active.

Sparse coding simply means that most cells have zero value - they are inactive [18]. In Hawkins & Ahmad’s HTM model, the few active cells jointly represent not just the current input, but previous inputs as well. This is achieved by applying local inhibition to groups of cells - within each group, different cells fire in response to different sequential contexts. Since a long context is encoded in the current state without filtering for salience, we will refer to this strategy as the memorization approach.

II Method

We will attempt to implement the memorization approach to sparse sequence encoding using a more conventional RNN, trained with stochastic gradient descent. For convenience we will refer to our approach as recurrent sparse memory (RSM). RSM is a predictive autoencoder derived from the sparse autoencoders developed by Makhzani and Frey [19, 20]. In their algorithm, a fixed top-$k$ sparsity is applied; sparseness is the only nonlinearity.

An autoencoder learns to reconstruct the current external input $\mathbf{x}^{F}(t)$ through a bottleneck. To make an autoencoder learn transitions, we instead train it to generate a prediction $\mathbf{\hat{x}}^{F}(t)$ of the next external input $\mathbf{x}^{F}(t+1)$ using a mean square error loss.

To allow the network to use previous inputs as context for the prediction, we add a recurrent input $\mathbf{x}^{R}$, containing a copy of layer cell activity from the previous step - similar to an Elman RNN architecture.

In some experiments we use more than one RSM layer. In this case we require an additional input from the adjacent higher layer $\mathbf{x}^{B}$. The superscript identifiers $F,R,B$ stand for feed-Forward, Recurrent, and feed-Back respectively.

Cells within the layer are organized into groups. All cells in a group share the same learned weights for external input $\mathbf{x}^{F}$. Each cell has its own set of fully-connected learned weights for recurrent input $\mathbf{x}^{R}$. The parameter $g$ specifies the number of groups. Parameter $c$ is the number of cells in each group.

II-1 Theory of operation

The network described above is similar to a conventional recurrent layer. However, we add an inhibition term $\phi$ (defined below) that causes cells to become selectively active in particular sequential contexts. The same observation in different contexts is then represented by a different set of active cells.

When a group of cells responds strongly to an input $\mathbf{x}^{F}$, the least inhibited cell in the group becomes active. Inhibition due to the refractory period ensures that all cells in a group learn unique contexts in which to fire. Fixed sparsity ensures that cells and groups learn to be active in a huge variety of unique combinations. Each combination of cells encodes not just the current input, but a long history of prior inputs as well. As in [17], the memory can represent many long histories using the current set of active cells, in effect learning a higher-order representation of these sequences.

II-2 Model Details

See Figure 1 for an overview of the model. We can calculate the matrix product of inputs and weights as normal for a single ANN layer. Note that each cell has a unique set of recurrent weights $w^{R}$, whereas all cells in a group share the same external input weights $w^{F}$:

[TABLE]

[TABLE]

[TABLE]

The weighted sum $\sigma_{ij}$ of a cell $j$ in group $i$ is given below. $\sigma$ is a matrix of dimension (groups $\times$ cells) i.e. $g\times c$:

[TABLE]

We shift the weighted sums to positive nonzero values and calculate cell activity $\pi$ after applying an inhibition term $\phi$:

[TABLE]

We reduce cell activity $\pi$ to group activity $\pi^{G}$ by taking the max value of the cells in each group:

[TABLE]

The next step is to calculate two sparse binary masks $M$. $M^{C}$ indicates the most active cell in each group. $M^{G}$ indicates the most active groups in the layer. We use a function $\operatorname{topk}(a,b)$ that returns a ‘1’ for the top $b$ elements in the last dimension of argument $a$, and ‘0’ otherwise.

[TABLE]

[TABLE]

We take the elementwise product of the weighted sum and the two masks to yield the weighted sum of one cell from each of the top-$k$ groups. This is the sparsening step. A nonlinearity is applied.

[TABLE]

Cells that are selected are inhibited in future, mimicking the refractory period observed in biological neurons. The refractory period also ensures good utilization of all cells during training. Inhibition decays exponentially, but since we use ranking to select groups, even tiny inhibitions can have a significant effect over long periods. The period of effective inhibition depends on the number of competing groups and cells available rather than the rate of decay. The hyperparameter $0\leq\gamma\leq 1$ determines the inhibition decay rate.

[TABLE]

Finally, we allow the option to integrate the recurrent input over time. $\epsilon$ determines the decay rate of the integrated encoding $\psi$. Since the current value of $\psi$ theoretically represents all previous states, when external input statistics are identical between training and test time, $\epsilon$ can be zero. However, we found a generalization advantage to nonzero $\epsilon$, discussed later.

[TABLE]

$\mathbf{x}^{R}$ is both the recurrent input and input to any task-specific classifier (see experiments, below). If more than one RSM layer is used, $\mathbf{x}^{B}_{lower}=\mathbf{x}^{R}_{higher}$ and $\mathbf{x}^{F}_{higher}=\mathbf{x}^{R}_{lower}$. $\alpha$ is a normalizing scalar such that the sum of $\mathbf{x}^{B}$ is 1 (since we have many zero values, and few or no negative values, we have not found a more appropriate norm). $\mathbf{x}^{R}$ is initialized as zeros and updated:

[TABLE]

Each RSM layer is trained to predict the next input $\mathbf{x}^{F}(t+1)$. Prediction $\mathbf{\hat{x}}^{F}$ is generated by “decoding” through a bottleneck of masked hidden layer group activities $\mathbf{y}^{G}$ by taking the max activity of the cells in each group $i$:

[TABLE]

$w^{D}$ is a set of decoding weights of dimension equal to the transpose of $w^{F}$. We observed that tied weights $w^{F}=(w^{D})^{T}$ were less effective in our experiments.

[TABLE]

The mean square error between prediction $\mathbf{\hat{x}}^{F}(t)$ and observation $\mathbf{x}^{F}(t+1)$ is minimized by stochastic gradient descent. Gradients propagate from the prediction through $w^{D}$ to the sparse bottleneck $\mathbf{y}^{G}$, and subsequently to the encoding weights $w^{F}$, $w^{R}$ and $w^{B}$. The latter two learn to modulate hidden activity given the sequential context. The backpropagation depth is fixed by this architecture to 2. We conceptualize an RSM layer as a single layer of neurons with two or three independently integrated dendrite trees. Max and ranking functions respectively represent local and regional competition between cells.

The architecture is extremely memory efficient, requiring only a single copy of prior memory state $\mathbf{x}^{R}$ and inhibition $\phi$ in addition to current external input $\mathbf{x}^{F}$ for both training and encoding. If $c$ is the number of cells, the asymptotic measure of RSM memory use is $O(c)$. Both truncated BPTT and feed-forward autoregressive approaches such as WaveNet [6] require $O(ct)$ where $t$ is the time horizon. With the exception of top-k ranking, all the functions described above are fast, simple arithmetic operations.

II-A Experimental Architecture

For most of our experiments we use a single RSM layer. To exploit the memory for a discriminatory purpose, we add a “classifier” network of two fully-connected layers (see Figure 2). We use leaky-ReLU nonlinearities in these networks to reduce the accumulation of dead cells due to nonstationary RSM encoding. As per our rules, gradients do not propagate from the classifier into the RSM layer, or across the recurrent or feedback inputs. All layers are trained simultaneously & continuously using only the current input, satisfying the objective of learning using local and immediate credit assignment.

With the addition of the classifier, the architecture is reminiscent of “Reservoir Computing” models [lukovsevivcius2009reservoir], with the primary differences being the memory is trained via stochastic gradient descent (reservoir “memory” is typically not trained), having sparse activation to improve the encoding properties, and inhibition to recruit larger cell populations during training.

III Experiments

All experiments use the same architecture described above, with the exception that the classifier is replaced with a Generative Adversarial Network (GAN) [goodfellowgan2014] in the generative multi-step prediction task.111The codebase and a full list of hyperparameters for each experiment are publicly available on GitHub at: https://github.com/Cerenaut/rsm

Published results concerning a sparse memorization approach such as HTM [21] focus mostly on network robustness and do not enable easy comparison to more popular ANNs. We will attempt to validate the approach on a range of more easily contextualized tasks & benchmarks.

III-A Associating distant cause and effect

LSTM was specifically designed to enable learning of distant cause and effect, using gated memory cells. Hochreiter and Schmidhuber demonstrated LSTM using several example problems, including the Embedded Reber Grammar (ERG) [10]. Since they showed that ordinary RNNs were unable to solve the task, it is a good benchmark for demonstrating distant cause & effect learning.

The ERG has an initial fork (T,P), followed by the same distraction subsequence on both forks (see Figure 3). The final step (T,P) can only be predicted by remembering the pre-fork symbol. For added difficulty, symbols T and P also occur in the distraction sequence.

RSM enables its attached classifier to achieve over $99\%$ accuracy on a random sample from the grammar (see table I).

The ERG is not deterministic. It generates sequences of minimum length 9. The maximum length is unbounded, but to achieve 99% accuracy, all sequences of length $\leq 30$ must be predicted correctly. Without gated memory cells, simple RNNs trained with BPTT and RTRL fail to solve the ERG task [10]. Cui et al report that HTM achieved 98% accuracy [21].

We speculate that RSM achieves high accuracy by memorizing all the sequences frequently generated by the grammar. In a sample of 5000 sequences drawn from the grammar, we observed 601 unique sequences. We used $g=200$, $c=6$ and $k=25$ for this experiment. Since there are only 48 disjoint sets of 25 cells, memorization of 600 or more sequences implies that RSM has learned a combinatoric model.

RSM prediction is not confounded by stochastic sequences, but note that the statistics of training and test sequences are identical. Both the RSM layer and classifier components of the network achieve the same accuracy, but the classifier allows a specific label to be predicted, rather than an image of a label.

III-B Modelling partially observable, higher-order sequences

We have demonstrated that RSM is able to model non-deterministic sequences, but what about sequences that are only partially observable? Since an RSM layer is easily derived from a convolutional network layer, and can handle non-deterministic sequences, it is likely RSM is able to simultaneously learn sequential and spatial structure in its input. To test this hypothesis, we presented repeating sequences of MNIST images and measured next-digit appearance and label prediction accuracy. Each step in a sequence has a specific label, but a random image of that label is selected to represent it as input. Due to variation in digit images, the actual sequence is only partially observable. True labels are never observed by the RSM. There are no sequence boundaries and the network is never reset to a baseline state, so all structure must be inferred from the stream of input.

Table II shows some of the sequences tested and the minimum history necessary for perfect prediction. Note that some sequences have Markov order $>1$, meaning that correct recognition of the current image is insufficient to predict the next label.

As before, both RSM and classifier components achieve accuracy of 99% or better, but the classifier allows labels to be obtained for quantitative measurement. Examination of the next-image predictions is interesting - the RSM generates generic digit-image predictions (see Figure 4).

When trained to classify the label of the current image, the same fully-connected classifier network achieves a lower accuracy of 98%. We conclude that the RSM is able to combine spatial and higher-order sequential information in a single layer to boost image classification accuracy to over 99% for all sequences tested. In many network architectures incorporating gated memory cells, a convolutional stack is used for spatial dimensionality reduction before consideration of prior state held in memory cells (for example, Srivastava et al [22] use a pretrained stack to generate “percepts” as input to an LSTM layer). Integrated modelling of spatial and temporal data in a single layer is likely advantageous, and has been investigated [23], but this line of research is hampered by the complexity of gated memory layers.

A recent publication by Gordon et al [24] applied a modified version of RSM to a similar task with the additional difficulty of non-deterministic sequences. In this case the modified bRSM was observed to learn the sequence structure better than an LSTM with the same number of trainable parameters.

III-C Multi-step video prediction

We wished to explore whether RSM can learn temporal dynamics well enough to generate novel sequences, rather than simply next-step predictions. Models trained to perform single-step prediction tend to perform poorly when used for multi-step, self-looped prediction because they lack a persistent representation of the whole production, such as a sentence [25]. In addition, we wanted to explore a task with more demanding spatial characteristics rather than a single character centered in the image. Multi-step video prediction is an obvious candidate task and current models are unsatisfactory [26].

The “Bouncing balls” video prediction task was introduced by Sutskever et al [27]. The objective is to predict the motion of three billiard balls bouncing around in a box. Interactions between balls and the walls of the box are not precisely predictable from the raster image observations provided as input. Researchers evaluate models trained on this dataset in two ways - first, by measuring next-frame prediction mean-square-error (MSE). Second, qualitatively by generating videos of the learned model in self-looped mode. In this condition, a few “priming” frames are produced by the simulation, and passed as input to the learned model. The learned model continuously generates predictions of the next input frame. After priming, the model’s own prediction is provided as input. Since tiny angular errors compound rapidly over time, it is expected that sequences generated in this way diverge from the simulation, making an MSE comparison inappropriate (Figure 6).

III-C1 Stacked RSM networks

The recurrent input $\mathbf{x}^{R}$ can also be utilized as $\mathbf{x}^{F}$ for a deeper RSM layer, allowing layers to be stacked. In the stack, credit assignment is still local within each RSM layer (no gradients propagate recurrently or between layers). Stacking up to 3 RSM layers and using the deepest layer as classifier input did not degrade performance in our image sequence experiments, suggesting that RSM layers can be assembled into a hierarchy. In the bouncing balls task, adding a 2nd RSM layer improved prediction accuracy significantly (see table III). Since a single RSM layer can learn higher-order models, there is little advantage to deeper networks unless it enables more efficient dimensionality reduction, or association of multi-modal inputs.

III-C2 Convolutional RSM

Given the higher spatial complexity of the bouncing balls task, a convolutional approach is desirable. A convolutional version of RSM can be easily derived from the fully-connected version given above. Learned parameters are shared between all convolutional filter positions, but inhibition $\phi$ and outputs $\psi$ are unique to each position. The number of filter positions (and hence, output dimension $[h^{1},w^{1}]$ of a lower layer) is determined by the feed-forward convolution operation. In a convolutional stack architecture, the number of filter positions in the higher layer $[h^{2},w^{2}]$ may be dissimilar to the lower layer $[h^{1},w^{1}]$. A mini-batch of convolution produces a matrix of dimension $[b,h,w,g,c]$. These dimensions are respectively, batch-size, conv. height, conv. width, groups and cells. For the feedback input $\mathbf{x}^{B}$ from higher layers, we set the receptive field width and height to $1\times 1$ and interpolate from $[h^{2},w^{2}]$ to $[h^{1},w^{1}]$. This ensures that $[h^{1},w^{1}]$ is equal to the number of filter positions required by the feed-forward convolution operation. A $1\times 1$ receptive field is also used for the recurrent input, meaning that each filter position only receives a recurrent input from itself.

III-C3 GAN rectifier

We observe that RSM generates a prediction covering all possible next input states (e.g. Figure 4). Each pixel prediction approximates the expected value of that pixel. When self-looped, the uncertainty in these predictions compounds over time, resulting in degraded input dissimilar to the training input. We noticed that using rudimentary image processing filters to rectify the predictions significantly improved RSM output in self-looped mode. A Generative Adversarial Network (GAN) is well suited to this role, because given a conditioning input it can produce clear, sharp samples that are indistinguishable from genuine input. We trained a GAN to take the lower-layer RSM prediction $\mathbf{\hat{x}}^{F}$ as conditioning input and output a next-frame sample. Implicitly, the role of the RSM is to predict the distribution of next frame samples, and the role of the GAN is to render one sample from this distribution. The GAN does not have access to any context to allow it to perform a predictive role. Gradients from the GAN do not propagate into the RSM layers.

We used a stacked convolutional autoencoder as the GAN generator, and a fully-connected network as GAN discriminator. The GAN was trained with both MSE and adversarial losses, copying the Predictive Generative Network (PGN) of Lotter et al [30]. However, the PGN architecture combines CNN and LSTM architectures to add the predictive aspect. In our case we rely entirely on the RSM layers for prediction.

III-C4 Video prediction results

Table III shows comparative performance of RSM and works by other authors. For reference, we include results for RSM without GAN and a single layer RSM + GAN. A two layer RSM + GAN system outperforms all prior works, including a variety of LSTM architectures, in terms of next-frame prediction accuracy. To better understand the generated spatio-temporal dynamics in self-looped mode, we recommend viewing a video of the output 222For a qualitative understanding of the generated ball dynamics, see video at: https://www.youtube.com/watch?v=-lBFW1gbokg. Although videos of self-looped output are not available for all works, the results are markedly improved over Sutskever et al (RTRBM / SRTRBM) [27]. Whereas the balls wander, change direction or acceleration and stick to each other in the latter, the RSM videos show relatively good conservation of momentum and plausible interactions between objects.

III-D Language modelling: Next word prediction perplexity

III-D1 Generalization

RSM takes the approach of remembering and encoding a long history of prior states in the current state. To achieve this, the same input in different sequential contexts generates highly orthogonal encodings. This is in contrast to gated memory layers, where a few features are selectively remembered, and different sequential contexts can generate similar encodings. Consequently, we anticipate that changes in sequential context between training and test datasets will severely disrupt RSM. In other words, we don’t expect it to generalize well to unseen sequences.

To explore this aspect of generalization, we selected a language modelling task. The Penn TreeBank (PTB) dataset is a corpus of approximately 1 million training words and 80,000 test words. We used Mikolov’s preprocessing [31], which results in a dictionary of 10,000 unique words. PTB is known to present a difficult generalization challenge due to the small size of the corpus. Next-word prediction quality is typically measured using perplexity (PPL). Model output is a distribution over the words in the dictionary. In our case, model input is a binary vector representing the (random) indices of each word in the dictionary.

III-D2 Generalization features

For RSM, even small changes between training and test sequences can be disruptive. Imagine that the training corpus contains ‘The cat sat on the mat’ and the test corpus ‘The cat sat on a mat’. Although the RSM encoding would be identical until ‘on’, subsequent states would be highly orthogonal, so the classifier cannot predict ‘mat’ from combinations of active cells never previously observed. To mitigate these types of generalization error, we enabled some minor features.

First, we integrate feedback to the RSM using $\epsilon>0$. This is not needed to help RSM learn sequences, but it does help RSM generalize learned sequences. We apply dropout at rate $0.5$ to encourage the RSM to make use of earlier encodings, for example the active cells representing ‘The cat sat’ are all predictive of a future ‘mat’. Second, we provide the integrated RSM state to the classifier. These changes reduce test perplexity by 10-15.

The third change is to “forget” the recurrent state of the network with a fixed probability $\mu$ during training. If forgetting is randomly selected, we set integrated activity $\psi$ and inhibition $\phi$ to zeros. The intention is to expose the classifier to different subsequences, in effect augmenting the training set. This feature improves test perplexity by another 5-10 points.

III-D3 Regularization

We attempted to regularize the RSM and classifier networks via conventional techniques - in particular adding an L2 loss term, and dropout. We found neither helpful when added to the RSM, but a small L2 value improved classifier performance.

To understand why these techniques are ineffective with this architecture, consider Cui et al [21], which was one of the inspirations for our work. They show that sparse distributed representations are extremely robust to network damage, because the meaning of individual elements is correlated. Therefore, to successfully “ignore” an observation, many cells’ activity must be simultaneously excluded. This is difficult for reasonable levels of both dropout and weight penalties. So it seems ineffective regularization may be a consequence of distributed representations that are robust to damage.

III-D4 Model interpolation

Mikolov et al [31] showed that linear interpolation of model distributions to produce ensemble predictions is very often effective at reducing test perplexity. In particular, earlier RNN and LSTM language models (which we are using as a baseline) were often presented as ensembles. 5-gram language models with Kneser-Ney smoothing (KN5) [32] are optimal for the PTB corpus; higher n-gram models do not deliver better test perplexity.

When a word occurs, it is likely to occur again in the near future. This general principle leads to cache language models. Mikolov et al[31] also showed that caches invariably improve ensemble perplexity. We felt that adding a fixed, exponentially-decaying probability mass to words after observation - a primitive cache - meets our biological plausibility criteria. We did not allow ‘adaptive’ modelling (i.e. allowing the RSM to learn during the test). We report RSM results interpolated with the simplistic cache, and optionally with KN5 for easier comparison and insights on n-gram ensemble complementarity.

III-D5 Results

With $g=600$ groups, $k=20$, $c=6$ cells per group and 4 epochs of training with batch size 300, we obtained 50% next word prediction accuracy and a perplexity of 9 on the training corpus. Training perplexity and accuracy had not plateaued. This suggests that moderately sized RSM layers can memorize extremely long sequences.

We found test corpus perplexity to be better with $n=8$ cells per group and only 0.25 epochs of training. We observed next word prediction accuracy of 20.6% on the test corpus. RSM per-word perplexity is bi-modal, showing rapid oscillations between very low and high values. When not combined with KN5, a small uniform mass significantly reduces average RSM test perplexity (reported result without KN5 includes 7% uniform mass). The complementarity of RSM and KN5 - evidenced by RSM+KN5 interpolated model perplexity - suggests that RSM sometimes captures longer range context, but also fails to generalize many shorter range contexts.

Table IV shows PTB test perplexity for RSM and selected popular language models. RSM is comparable to some early methods such as 5-gram Good-Turing, Random Clusterings, Random Forest and 5-gram Maximum Entropy, both individually and interpolated with KN5. The natural point of comparison is conventional RNNs trained with short-context BPTT. RNNs trained with BPTT are significantly better than RSM, with PPL 124 (RNN) and 105 (RNN+KN5).

Carefully regularized, gated-memory ANN language models such as LSTM reduce PTB test perplexity from 124 to 100 or less [35], with state of the art scores $<60$ [33]. Models with attentional filtering are even better [34]. However, these improvements required several years’ effort by the research community, and exploit selective remembering features that could also be added to RSM over time.

Gordon et al [24] recently published a modified version of RSM (bRSM) that replaces inhibition with a boosting scheme. The resultant PTB perplexity of 103 is significantly better than RNNs trained with BPTT (PPL 124) [31]. This suggests that bRSM overcomes most or all of the generalization issues demonstrated by RSM.

IV Summary

Our primary objective was to develop a strictly local, biologically-plausible credit assignment strategy with comparable practical performance to early deep-backpropagation results. We believe we have been successful in this regard. On several tasks RSM provides results comparable to neural methods trained by deep BP or BPTT, which should not be possible for a simple RNN. For example, we showed that RSM can successfully learn to exploit distant causes when predicting later effects in the ERG task, which an RNN without BPTT cannot do, and in fact requires gated memory such as LSTM [10]. RSM can learn higher-order sequences, likely with combinatoric representational efficiency, despite some degree of sequential uncertainty and partial observability. On the “bouncing balls” task, RSM produced new best results beating a range of algorithms trained by deep BP, including LSTM. The learned model is also convincing in self-looped mode, generating arbitrarily long sequences of quite realistic billiard-ball dynamics.

RSM has a number of advantages - an order of magnitude reduction in memory requirements, very large memory capacity, batch-online training, and it is not necessary to specify the maximum time horizon in advance. For complex sequence learning tasks with similar training and test statistics, the RSM approach is appealing due to its simplicity and speed. However, although stochastic processes are not confounding, generalization to unseen test sequences with different statistics is clearly limited in its current form, as seen in the PTB task. A later development of RSM by Gordon et al [24] was able to overcome this limitation, giving better results on PTB than RNNs trained by BPTT. This reinforces the evidence that RSM represents a viable alternative credit assignment strategy to deep backpropagation through time.

In the image sequence prediction task, RSM was continually exposed to unseen sequences of observations due to the large number of images of each digit. Why was this form of generalization not pathological? First, the memorization approach is unlikely to confound spatial generalization; it is only unseen sequence generalization that is limited. Second, it is not possible to memorize the infinite observed training sequences, and the training regime therefore forces RSM to learn a generalized sequence model. Since both training and test set are drawn from the same distributions, this form of generalization is perhaps only interpolation rather than extrapolation.

V Future work: Selective remembering

One of the most significant differences between RSM and gated memory layers such as LSTM is the strategy of “remember everything” versus “selectively remember useful things”. The latter approach is able to generalize better, because it is not distracted by irrelevant features in unseen test data: selective remembering can make different contexts become similar.

Selective remembering can also be considered a form of attention. Recent work has shown that self-attention is a very effective mechanism for selecting useful features from sequences [36], with groundbreaking results in neural language models. Radford et al [34] use self-attention in an unsupervised next-word-prediction training regime, like RSM, but autoregressive, feed-forward, rather than recurrent.

In future work we will attempt to use self-attention to selectively remember (sub) sequences using RSM, without violating our biological plausibility constraints.

Acknowledgment

The authors would like to thank Alan Zhang, for conversations about the nature of generalization and learning representations.