Locally Connected Spiking Neural Networks for Unsupervised Feature Learning

TL;DR

This paper introduces Locally-Connected Spiking Neural Networks (LC-SNNs) that learn image features using STDP, achieving state-of-the-art accuracy with fast convergence, fewer parameters, and robustness to synapse and neuron deletion.

Contribution

The paper presents a novel LC-SNN architecture with biologically inspired local connectivity and competitive learning, outperforming previous SNN models in unsupervised image classification.

Findings

Achieved state-of-the-art accuracy on two image datasets.

Fast convergence with fewer learnable parameters.

Robust performance despite synapse and neuron removal.

Abstract

In recent years, Spiking Neural Networks (SNNs) have demonstrated great successes in completing various Machine Learning tasks. We introduce a method for learning image features by \textit{locally connected layers} in SNNs using spike-timing-dependent plasticity (STDP) rule. In our approach, sub-networks compete via competitive inhibitory interactions to learn features from different locations of the input space. These \textit{Locally-Connected SNNs} (LC-SNNs) manifest key topological features of the spatial interaction of biological neurons. We explore biologically inspired n-gram classification approach allowing parallel processing over various patches of the the image space. We report the classification accuracy of simple two-layer LC-SNNs on two image datasets, which match the state-of-art performance and are the first results to date. LC-SNNs have the advantage of fast convergence to a dataset representation, and they require fewer learnable parameters than other SNN approaches with unsupervised learning. Robustness tests demonstrate that LC-SNNs exhibit graceful degradation of performance despite the random deletion of large amounts of synapses and neurons.

Figures

Tables

Table 1. Table 1 : Test accuracy results for LC-SNNs with varying numbers of n filters subscript 𝑛 filters n_{\textrm{filters}} using the 2 2 2 -gram classification scheme. Each result is estimated from 10 independent train and test phases.

$n_{\text{filters}}$	$(k,s)$	$n_{\text{synapses}}$	$n_{\text{neurons}}$	mean $\pm$ std.
25	(12, 4)	32.4K	225	84.01 $\pm$ 2.28
50	(14, 4)	64.8K	450	88.21 $\pm$ 1.79
100	(12, 4)	129.6K	900	91.68 $\pm$ 1.31
250	(16, 2)	576K	2250	93.71 $\pm$ 0.68
500	(16, 2)	1.15M	4500	94.59 $\pm$ 0.61
1000	(16, 2)	2.3M	9000	95.07 $\pm$ 0.63

Table 2. Table 2 : Comparison of SNN training results on the MNIST dataset. See respective papers for details on SNN architectures and learning procedures.

Paper	Architecture	Learning	mean	std.
Ours	Local + recurrent connections	Unsupervised	95.07	0.63
[23]	Dense + recurrent connections	Unsupervised	95.00	-
[29]	Input segmentation	Unsupervised	75.93	-
[26]	Dense + recurrent connections	Incremental unsupervised	86.59	-
[27]	Autoencoder	Semi-supervised	99.08	-
[28]	Dense + lateral connections	Supervised / semi-supervised	95.4 / 72.1	-
[31]	Convolutional	Semi-supervised	96.95	0.08
[32]	Convolutional	Semi-supervised	99.28	$\approx$ 0.1
[33]	Convolutional	Semi-supervised	97.20	0.07
[35]	Convolutional	Semi-supervised	98.4	-

Table 3. Table 3 : Test accuracy results for LC-SNNs on the EMNIST “letters” partition with varying numbers of n filters subscript 𝑛 filters n_{\textrm{filters}} using the 2 2 2 -gram classification scheme. Each result is estimated from 10 independent train and test phases.

$n_{\text{filters}}$	$(k,s)$	mean $\pm$ std.
25	(12, 4)	49.61 $\pm$ 1.98
50	(12, 4)	54.44 $\pm$ 2.04
100	(12, 4)	62.34 $\pm$ 2.11
250	(12, 4)	65.47 $\pm$ 1.73
500	(12, 4)	67.58 $\pm$ 1.78
1000	(16, 2)	69.73 $\pm$ 1.57

Table 4. Table 4 : Test accuracy results for LC-SNNs with different classification schemes. Each result is for n filters = 100 subscript 𝑛 filters 100 n_{\textrm{filters}}=100 and ( k , s ) = ( 12 , 4 ) 𝑘 𝑠 12 4 (k,s)=(12,4) .

Method	Accuracy
Single thread: 2-gram	$91.68$
Multi-thread: 2-gram	$88.91$
Multi-thread (Sum of spikes): 3 votes	$93.21$

Equations

$${\tau }_v\frac{\text{d}v(t)}{\text{d}t}=-v(t)+v_{\text{rest}}+I(t),$$

$${\tau }_{\theta }\frac{\text{d}\theta (t)}{\text{d}t}=-\theta (t),$$

$$\Delta w=\{\begin{array}{ll}{\eta }_{\text{post}}{x}_{\text{pre}}& \text{on post-synaptic spike;}\\ -{\eta }_{\text{pre}}{x}_{\text{post}}& \text{on pre-synaptic spike.}\end{array}$$

Full text

††journal: Neural Networks

1 Introduction

Processing of image data by machine learning (ML) systems often requires techniques for overcoming the problem known as the curse of dimensionality; that is, the inherent difficulty in processing high-dimensional data. The locally-connected layer of the neural networks literature [1] imposes an inductive bias on an artificial neural network (ANN) that stipulates that certain features only occur in particular regions of input space. This is in sharp contrast to the popular convolutional layer, in which a set of filters are learned that can be used to detect features across the entire input space, inducing a translation invariant data representation [2]; i.e., in which the occurrence or relative locations of features are important. In the deep learning (DL) literature, both methods were key to reducing the number of parameters needed to train deep and relatively efficient networks for processing complex data; e.g., images and videos [citations needed].

Spiking neural networks (SNNs) [3] are well-known for their energy efficiency and speed on neuromorphic hardware platforms [4]. For this reason, development of methods that enable SNNs to reach the performance of ANNs on machine learning tasks has gained interest in recent years [5]. Previous work has shown that spiking neural networks can, in principle, be used for performing complex machine learning tasks; e.g., image classification [6, 7] and reinforcement learning [8, 9]. On the other hand, few methods exist for the robust training of SNNs from scratch for general-purpose machine learning ability; i.e., their abilities are highly domain- or dataset-specific, and require much data pre-processing and tweaking of hyper-parameters in order to attain good performance.

This paper aims to extend the abilities of spiking neural networks by introducing and studying a spiking version of a computational layer borrowed from the artificial neural networks literature. In particular, the locally connected layer is introduced and implemented in spiking neural networks, to which arbitrary biological learning rules (e.g., Hebbian learning, spike-timing-dependent plasticity, etc.) may be applied [10]. Simple two-layer networks are built and trained in an unsupervised fashion to produce a representation of the MNIST [11] and EMNIST [12] datasets, and ultimately, the goodness of the representation (i.e., to what extent the learned features linearly separate the classes of data) is assessed through a simple voting mechanism on the spiking outputs of the trained networks. In particular, we use the $n$-gram method of [13]. The introduced spiking network component and corresponding training procedure produce distributed representations which are robust to a large degree of randomly deleted synapses or neurons, a property commonly referred to as graceful degradation.

There are various popular libraries in the literature that allow efficient spiking neural network simulations. Some of them provide deep elaboration of biological details; see, ANNarchy [14], BRIAN [15], Genesis [16], NEURON [17], and NEST [18]. Other approaches focus on high-level behaviors of spiking neural networks, with benefit for machine learning applications, e.g., NeuCube [19], and NENGO [20]. We developed the spiking neural network software package BindsNET written in Python [21]. BindsNET provides us with significant freedom in deciding implementation details, aiming at machine learning applications. BindsNET builds on the results of [22] and [23], wherein all networks were implemented in the BRIAN spiking neural networks simulator [24]. In addition to image classification tasks addressed here, BindsNET has been used implementing SNNs playing computer games such as Atari breakout, by transferring weights from Deep Q-Learning NNs trained by reinforcement learning [25]. The SNNs introduced in this paper are implemented in BindsNET and have been trained by STDP algorithm to solve image recognition tasks. The BindsNET library enables the simulation of networks on CPUs or GPUs or a combination thereof, and GPU computation is useful in particular for the fast simulation of large LC-SNNs with many neurons and synapses.

2 Related work

Due to their potential for efficient implementation on neuromorphic hardware devices, much recent work has focused on developing powerful and efficient learning algorithms for SNNs. We focus on SNNs trained for machine learning tasks in simulated environments, which may be either (1) converted to neuromorphic platforms post-training, or, in some cases (2) trained and evaluated entirely on neuromorphic chips. A review of machine learning with SNNs is treated by [5].

This work is related most closely to that of Diehl & Cook [23], in which a simple three-layer network is trained unsupervised with spike-timing-dependent plasticity along with excitatory-inhibitory interactions between neurons to learn to classify the MNIST handwritten digits [11]. The input layer is comprised of Poisson spiking neurons fully-connected with STDP-modifiable synapses to an arbitrarily-sized layer of excitatory neurons. Neurons in this layer “compete” with one another to learn filters that capture prototypical examples from the data via connections from an inhibitory layer. Once trained, this network operates similarly to a winner-take-all (WTA) circuit, with sparse activity in the excitatory population coding for a single category of data. Several extensions of this work has been explored quite recently, including the simultaneous clustering and classification of image data [13] and the incremental learning of distinct categories [26].

Several other unsupervised learning methods have been developed and demonstrated with SNNs. A network inspired by the autoencoder of the neural networks literature is trained without labels layer-wise to reconstruct the MNIST and CIFAR-10 datasets, and whose output is trained in a supervised fashion to perform classification [27]. A learning algorithm for SNNs is developed on the supposition that neurons tend to maximize their activity in competition with other neurons in [28]. Three-layer networks of Izhikevich regular spiking neurons are trained with STDP on the binary task of distinguishing the digits 0 and 1 as well as on the full set of MNIST digits [29].

In this paper, we are primarily interested in the implementation of locally-connected layer (see, e.g., [1]) for spiking neural networks. A closely related concept is the convolutional layer [30], which is named for how it convolves a number of weight kernels across the input space while the weights remain fixed, or shared between locations in the input space. A number of studies have implemented convolutional layers for SNNs, with a mixture of supervised and unsupervised learning methods [31, 32, 33, 34, 35]. Our locally connected layer is similar to this concept, except that a different set of weight kernels is learned per input location; i.e., the weights are not shared as we stride across the input space.

3 Methods

3.1 Spiking neuron model

Our networks use a variant of the popular leaky integrate-and-fire (LIF) spiking neuron with adaptive firing thresholds. The dynamics of the LIF voltage $v(t)$ is given by

[TABLE]

where $\tau_{v}$ is the membrane time constant, $v_{\textrm{rest}}$ is the neuron’s resting voltage (to which it decays exponentially in time), and $I(t)$ denotes the total input to the neuron at time $t$. The voltage (membrane potential) $v(t)$ of a post-synaptic neuron is increased at the occurrence of a spike from a pre-synaptic neuron connected to it by the weight $w$ of the synapse that connects them; all such incoming currents are gathered in $I(t)$. When a neuron reaches or exceeds its threshold membrane potential $v_{\textrm{thresh}}(t)$, it emits a spike to downstream neurons and resets to a voltage $v_{\textrm{reset}}$. At this point, the neuron is clamped to the reset voltage for a short refractory period $\Delta_{\textrm{ref}}$ and does not integrate its inputs.

The adaptive threshold $\theta(t)$ is used to ensure that a single neuron does not end up dominating the firing activity of the output layer, and that different neurons tune selectively to the intensity of different prototypical inputs. It has dynamics

[TABLE]

where $v_{\textrm{thresh}}(t)=\theta_{0}+\theta(t)$, with $\theta_{0}>v_{\textrm{reset}},v_{\textrm{rest}}$ a constant. Each time a neuron fires a spike, the adaptive threshold $\theta$ is increased by a constant $\theta_{\textrm{plus}}$.

In our experiments, we set $\theta_{0}=-52$mV, $v_{\textrm{rest}}=v_{\textrm{reset}}=-65$mV, $\Delta_{\textrm{ref}}$ = 5ms, $\tau_{v}$ = 20ms, $\tau_{\theta}$ = 1000s, and $\theta_{\textrm{plus}}$ = 0.05. In the context of machine learning, it may be of interest to include these in a grid- or random-search hyper-parameter optimization algorithm, but for the experiments described in this paper, they remain fixed.

Input neurons (used to convert non-negative image data to spikes in our experiments) are modeled by Poisson spikes trains with average firing rate equal to the input intensity in Hertz. For the experiments in Section 4, we multiply the MNIST and EMNIST images (having grayscale values in [0, 255]) by a factor of $\frac{1}{2}$ to constrain input firing rates from 0Hz to 127.5Hz.

3.2 Learning rule

Our locally-connected SNN layers are trained with a simple spike-timing-dependent plasticity (STDP) rule combined with a weight normalization scheme. The STDP rule combines both pre- and post-synaptic weight updates in order to detect temporally coincidental spikes from connected neurons. Our STDP rule has the form:

[TABLE]

The parameters $\eta_{\textrm{pre}},\eta_{\textrm{post}}$ are pre-, post-synaptic learning rates, respectively, and the quantities $x_{\textrm{pre}},x_{\textrm{post}}$ are pre-, post-synaptic spike traces, which are set to 1 on (pre-, post-synaptic) spike events, and are exponentially decaying to 0 otherwise. The spike traces are a simple memory trace of a neuron’s previously-occurring spike, used for implementing STDP in an online fashion; i.e., carried out during the SNN simulation. In our experiments, we set $\eta_{\textrm{post}}>>\eta_{\textrm{pre}}$ to emphasize the effect that pre-synaptic spikes have on post-synaptic neuron membrane potentiation. In particular, we used values around $\eta_{\textrm{post}}=1\times 10^{-2}$ and $\eta_{\textrm{pre}}=1\times 10^{-4}$, but these (and their exponential decay rate) were subject to grid search in all experiments.

Due to the unbalanced nature of the pre- and post-synaptic STDP updates, another mechansim is required to ensure that synaptic weights don’t grow without bound. First and foremost, weights are clipped after each iteration to the range $[0,w_{\textrm{max}}]$, and secondly, the sum of weights incident to a post-synaptic neuron is normalized after each iteration to have sum $c_{\textrm{norm}}$ (normalization constant). That is, if there are $n_{\textrm{pre}}$ pre-synaptic neurons, the synapses incident to a single post-synaptic neuron have average value ${c_{\textrm{norm}}}/{n_{\textrm{pre}}}$.

3.3 Network architecture: locally-connected SNN

The locally-connected layer is borrowed from the deep learning literature [1] and implemented in a spiking neural network. We restrict our discussion to data with two spatial dimensions; e.g., image data, with or without multiple color channels. At its most basic, the layer requires parameters $k_{1},k_{2}$ (horizontal and vertical kernel size, respectively), $s_{1},s_{2}$ (horizontal and vertical stride length, respectively), and $n_{\textrm{filters}}$, the number of filters to learn per input location (receptive field). Locations in the input space are determined by the kernel size and stride parameters; the first receptive field resides at the upper left corner of the input space, with spatial extent $k_{1}\times k_{2}$, and the filter is slid over by $s_{1}$ pixels until the right edge of the filter is incident to the right edge of the image, at which point the filter is moved down $s_{2}$ pixels and back to the left edge of the image, and process starts anew. The process is repeated until the entire input space is tiled with filters.

The SNN implementation of a locally-connected layer simply stipulates that there are synapses from the pre-synaptic to post-synaptic populations which coincide with the location of filters in the pre-synaptic population, and where the size of the post-synaptic population is determined by the parameters ($k_{1},k_{2},s_{1},s_{2},n_{\textrm{filters}}$) of the layer. In this paper, we restrict attention to the case $k_{1}=k_{2}$ and $s_{1}=s_{2}$ to simplify parameter search, and refer to the parameters respectively as $k$ and $s$.

We refer to a network which contains a locally-connected layer as a locally-connected spiking neural network (LC-SNN), although the network may contain additional layers with other layers included. Locally-connected layers are not restricted to receiving connections directly from the input; they may appear in any sequence of processing in a SNN. For example, multiple locally-connected layers can be connected in sequence to detect progressively more complex features in a hierarchical fashion.

In this work, we report classification results from a two-layer (input, output) LC-SNN. A schematic of this architecture is shown in Figure 1. An input example is rate-coded as the average firing rates of Poisson spiking neurons, which are connected with locally specific, STDP-modifiable synapses to the output layer. The output layer neurons are connected to other neurons which share the same receptive field with fixed inhibitory synapses. This inhibitory connectivity forces neurons in the output to operate in a WTA-like fashion, where, ultimately, one neuron per receptive field remains firing after all others are inhibited by its activity. This is similar to the fully-connected SNN architecture of [23], except that the input population is divided into a number of (possibly overlapping) sub-regions, instead of connecting the full input population to every output neuron.

3.4 Voting mechanism

In a commonly used voting approach, each individual output spike counts as a single vote, during the observation period. The more votes a neuron gets, the more likely it is selected. In the max decision rule, the winner will be the neuron with the highest number of spikes (votes). In this work, we explore the $n$-gram method as well to classify new input examples based on historical (training) firing activity [13]. We set $n$ = 2 for the sake of computational expediency. Accordingly, we record all length 2 sequences of spikes from the output layer during the training phase. These sequences are aggregated over the training phase, and are paired with the class of data for which they appear the most. When a test example is presented, all length 2 sequences are recorded, and each gets a vote (based on the label they’re paired with) towards the classification of the new example.

In this way, we are capturing simple information about the ordering of firing and harnessing it to distinguish the label of the data. This method is a step beyond a typical voting approach mentioned before, in which individual output spikes count as single votes. In the n-gram approach, we are taking advantage of the time domain aspect of spiking neural networks by supposing that the ordering of firing matters; i.e., permuting the order of output spikes results in a loss of information from the perspective of network encoding. In this work, we report only results using the $n$-gram method, but note that this method outperforms our experiments with single spike, vote-based (and re-weighted vote-based) classification schemes.

3.5 Model selection

A problem common to many machine learning tasks is picking the best version of a trained model to use for the test phase. To this end, during the training, the test accuracy of the network is periodically estimated by evaluating it with the $n$-gram classification scheme on a random sampling of examples from the training dataset. The version of the network which obtains the highest estimated accuracy is saved to disk, to be tested on the full test dataset. In our experiments, we estimate the test accuracy after every 250 examples on a randomly sampled batch of 250 examples.

4 Results

In the following sections, we present unsupervised learning results from LC-SNN networks with two-layer architectures, and show how training convergence of our methods improve over that of previous work on unsupervised learning with SNNs. These results build on those obtained by [22].

4.1 Two-layer LC-SNN results

We report test accuracy results from two-layer LC-SNNs with varying numbers of locally-connected filters $n_{\textrm{filters}}$. The network consists of an input layer of size equal to the input space (images center-cropped to 20 $\times$ 20), an output layer with size determined by parameters $(k,s,n_{\textrm{filters}})$, STDP-modifiable synapses from input to output layer, and a recurrent inhibitory connection in the output layer. As mentioned above, output neurons are inhibited by other output neurons that share the same receptive field in the input space.

4.1.1 MNIST digits dataset

Grid search is conducted to find the best setting of hyper-parameters in terms of classification accuracy, subject only to experimental time constraints. All experiments are limited to running for 24 hours with 12Gb of RAM. With large $n_{\textrm{filters}}$ and certain settings of $k$ and $s$, there is not enough memory or time to complete the simulation, further constraining the hyper-parameter search space. All networks are trained for 60K iterations; i.e., one pass through the MNIST training data, and tested on all 10K test examples. Both training and test examples (cropped to size 20 $\times$ 20) are presented to the network for 250ms of simulated time with a 1ms timestep. The results are reported in Table 4 along with the setting of kernel size ($k$) and stride ($s$) that resulted in the best accuracy per number of filters $n_{\textrm{filters}}$, as well as the number of STDP-modifiable parameters $n_{\textrm{synapses}}$ and output layer neurons $n_{\textrm{neurons}}$ those settings required, computed as a function of $k$, $s$, and $n_{\textrm{filters}}$.

The average test confusion matrix for the best set of hyper-parameters for LC-SNNs with $n_{\textrm{filters}}=1000$ is shown in Figure 2(a).

These accuracy results are comparable to similar work by [23], in which simple SNNs with a fully-connected layer are trained with STDP and competitive inhibitory interactions to classify the MNIST dataset. We refer to these networks from this point onwards as a baseline. However, our method is able to achieve higher test accuracy with fewer parameters due in part to the parameter-efficient local connectivity and improved training methods. On the other hand, the accuracy of methods is similar when compared instead by the number of neurons utilized. We plot a comparison of $n_{\textrm{synapses}}$ and $n_{\textrm{neurons}}$ versus test accuracy for the baseline SNN and the LC-SNN in Figure 3. By adding more locally-connected filters, we expect that LC-SNN test accuracy will exceed that of the baseline SNN’s best result.

Moreover, while we limit ourselves to a single pass through the training data, the results of [23] are reported with 1, 3, 7, and 15 passes through the training dataset for networks with 100, 400, 1,600, and 6,400 neurons, respectively. For a better comparison, we re-implemented the baseline SNN in BindsNET and ran a superset of the same experiments with a single pass though the training data. These results are also depicted in Figure 3, from which one can see that the larger baseline SNNs are severely undertrained without multiple passes though the training data. We expect that LC-SNN performance will further improve with multiple presentations of the training set.

Figure 3(a) demonstrates that, with a single pass through the training data, any size LC-SNN (in terms of STDP-modifiable synapses) achieves a better performance than the corresponding baseline network. On the other hand, Figure 3(b) show that smaller baseline SNNs attain better accuracy in terms of the number of spiking neurons required in the output layer. However, in our simulations, the main simulation bottlenecks were related to the dimensionality of the tesnors implicated in linear algebra operations, namely by the size of the weight matrices of the SNNs. To that end, the LC-SNN demonstrates a clear advanatage over the baseline in terms of simulation training time.

Figure 3(c) shows that the LC-SNN induce many more spikes than the baseline SNN, due to the distributed nature of the recurrent inhibitory connection; i.e., whereas the baseline networks operates similarly to a winner-take-all circuit, in which a single neuron typically causes all others to fall quiescent, the LC-SNN allows one neuron per receptive field to “win” in a similar manner. From a energy efficiency point of view, the LC-SNN implemented in neuromorphic hardware would be more costly than the baseline due to the increased activity, but with the benenfit of reduced training time and fewer synaptic connections. This can be seen as a tradeoff between energy and training time requirements, where the extreme sparsity of the baseline network’s activity results in a bottleneck in training convergence. Note that the number of LC-SNN spikes is only $\approx$ 2-5X greater than the baseline networks’ for all numbers of neurons considered.

Table 2 compares a collection of results from training spiking neural networks to classify the MNIST handwritten digits. We consider methods which explicitly use labels to learn all network parameters supervised, whereas methods which employ a partial unsupervised training are denoted semi-supervised. Of all considered unsupervised methods, our approach reaches state of the art results. On the other hand, semi-supervised and supervised methods are often superior, likely due to incorporating label information directly into the learning procedure.

4.1.2 EMNIST: letters partition

The partition of the EMNIST dataset containing the 26 capital letters of the English alphabet is used to once again test our LC-SNN approach against that of the baseline network architecture. Grid search is once again used to select the best setting of hyper-parameters (kernel size, stride, learning rate, and learning rate decay) per setting of $n_{\textrm{filters}}$. A single pass through the 124K training examples is used (4.8K per class) regardless of network size. The results are reported in Table 3.

For reference, grid search was performed over hyper-parameters of the baseline SNN, which obtained its best accuracy of 66.56% $\pm$ 1.96% with $n_{\textrm{neurons}}$ = 900. As with the LC-SNN, a single training epoch is used before the network is evaluated on the test partition. The best setting of LC-SNN hyper-parameters for $n_{\textrm{filters}}=500$ has 648K trainable synapse weights, comparable to the best baseline network with 705.6K parameters, while achieving better performance. Again, as the number of neurons in the output layer of the baseline network is increased, the number of passes through the training data needed to converge in performance increases dramatically. The average test confusion matrix for the best set of hyper-parameters for LC-SNNs with $n_{\textrm{filters}}=1000$ is shown in Figure 2(b).

To our knowledge, this work is the first to report results on the training of spiking neural networks to classify the EMNIST “letters” partition. Although we focus on only the 26 capital letters of the English alphabet, it remains to be seen if this approach will scale to all 62 categories of the original dataset.

4.2 Training convergence

A desirable feature of the LC-SNN is the rapid convergence speed which the distributed sub-network learning enables. That is, since only neurons within the same sub-population of the output layer (as determined by unique receptive fields) compete to learn filters, as many neurons as there are receptive fields can fire and subsequently update their synapse parameters at once. This leads to many more parameter updates than was previously possible, and a faster convergence to the network’s peak classification accuracy.

Moreover, this convergence does not appear to depend on the size of the network; i.e., the number of locally-connected filters. On the other hand, the baseline SNN networks requires more training iterations as the size of the network increases in order to reach their best performance. This phenomenon is depicted in Figure 4. Note that LC-SNNs often come with a few percentage points of their best accuracy within $\approx$ 5K training examples, whereas the baseline SNNs often require more than 60K examples to converge; e.g., for networks with 1,600 neurons, as demonstrated empirically in Section 4.1.1.

5 Robustness tests

In the following, we perform the following robustness tests on a fixed, trained LC-SNN:

Randomly deleting plastic synapses (from the input to the output layer) post-training. 2. 2.

Randomly deleting neurons from the output layer (equivalent to deleting all incoming synapses to said neurons) post-training.

We fix a single LC-SNN (with $k=14,s=2$, and $n_{\textrm{filters}}=250$) and baseline SNN (with $n_{\textrm{neurons}}=900$) which both achieved $\approx$ 93.5% test accuracy on the MNIST dataset, which are used in all robustness tests. The filters of the LC-SNN are re-shaped to be two-dimensional and visualized in Figure 6(a). Note that the baseline network required 3 passes through the training data in order to attain the same level of accuracy as the LC-SNN with only a single training epoch.

5.1 Deleting synapses

For each STDP-modifiable synapse in the aforementioned fixed LC-SNN, we delete it with probability $p_{\textrm{delete}}$, and run the test phase of the MNIST digits as normal. This gives us a sense of how performance degrades in the face of missing inputs; we run the test phase 5 times in order to get the average and standard deviation statistics of the performance result. The results are shown in Figure 5.

As can be seen in the figure, the network without any synapses deleted maintains an accuracy of $\approx$ 93.5%, while increasing $p_{\textrm{delete}}$ leads to a smooth degradation in performance. Interesting, with 50% of synapses removed, nearly 90% accuracy is maintained, and with 90% of synapses removed, nearly 60% accuracy is maintained. The baseline SNN drops off in performance at a faster rate than the locally connected network, possibly due to the distributed nature of the LC-SNN representation, as opposed to the fully-connected representation learned by the baseline.

An example deletion of synapses with $p_{\textrm{delete}}=0.5$ is given in Figure 6(b).

5.2 Deleting neurons

Similar to the approach in Section 5.1, we consider neurons in the output layer of the LC-SNN, and remove each with probability $p_{\textrm{remove}}$. We can implement this simply by setting all incoming synapses to a “removed” neuron to 0, effectively cutting it off from any positive input activity. This experiment assesses the degree of redundancy encoded in individual neurons, as opposed to the previous experiment which assessed the importance of individual synapses. The MNIST test phase is run 5 times to obtain mean and standard deviation statistics of the test performance result, which are depicted Figure 5.

From the figure, we conclude that the network filters encode a high degree of redundancy, as deleting large numbers of neurons results in relatively little reduction in accuracy. For example, deleting $90\%$ of neurons reduces accuracy down to just under $80\%$, reducing the number of filters down from 150 to approximately 15 per receptive field. The baseline SNN’s accuracy drops at a faster rate, again possibly due to the difference in nature of the two architecture’s learned representations. On the other hand, since the compared baseline SNN has fewer neurons, deleting a large percentage of them may have a more dramatic effect than for the LC-SNN.

An example removal of neurons with $p_{\textrm{remove}}$ is given in Figure 6(c).

6 Discussion

6.1 Multi-thread decision algorithm

Here we introduce an additional biologically-motivated method for deriving decisions from the LC-SNN spike trains. Previously results were obtained using the n-gram approach with very good accuracy. In the proposed multi-thread decision rule, we do not simply select the winning n-grams from all spike trains, but consider the location (patch) where the winning n-grams are coming from. In this way we contrast the temporal dimension of spike trains with their spatial distribution in multiple locations, representing multiple threads. Each of the patches form their own 2-gram dictionary and vote on the top N classes the ($1^{st}$ class gets the highest weight and the $N^{th}$ class gets the least weight). We also tested a multi-thread method of classifying based on the sum of spikes. Unlike the n-gram method, this method does not utilize the temporal aspect of the spikes. Based on the sum of spikes, patches associate each of their neuron with a different classes based on the training data of sum of spikes of the neuron. During classification, the patches vote on the top N classes. The overall decision is made by choosing the class that gets the most votes. This method of classification is much faster than the n-gram method as it does not require building n-gram and does not rely on the temporal aspect of the spikes.

The results with multi-thread algorithm applied to MNIST classification are summarized in Table 4. Note that the multi-thread sum of spikes method where each patch vote for top 3 classes is very good and it gets close to the logistic regression accuracy of $94.39\%$.

It is important to point out that the multi-thread approach has strong biological motivation and it may provide insights on the operation of biological mechanisms of spike processing. This is illustrated by Fig. 7, where the trained filters of the winning neurons are depicted for the single-thread 2-gram and multithread approaches. We can see that the multi-thread filters are in line with the expectation that a well-functioning processing system can efficiently extract information from various locations of the input data. This way the method has the promise of being scalable to increasing input data sizes.

6.2 Concluding remarks

We have demonstrated a new architectural building block, the locally-connected layer, for spiking neural networks and applied it to solve simple image recognition tasks. The layer allows two-layer networks to divide and conquer the input space by learning a distributed representation. Not only does this division of labor allow a reduction in the number of learnable parameters to achieve a certain quality of representation (as measured by classification accuracy), but it reduces the number of training examples required to converge to it. These LC-SNN properties have been empirically demonstrated on the MNIST and EMNIST datasets, where we have shown improved performance with shorter training times compared to a baseline, fully-connected SNN.

The locally-connected layer may be naturally included as a component in a multi-layer spiking neural network alongside with other feedforward, lateral, or recurrent connections. It encodes the inductive bias that features in the pre-synaptic space may be divided into possibly overlapping regions without loss of information, and for the added benefit of reduced parameter requirements and increased training convergence speed. Although the LC-SNN induces approximately 2-5 times more spikes during its operation than the considered baseline SNN, we argue that this is an acceptable tradeoff in light of the aforementioned convergence and accuracy properties. Indeed, we argue that the extreme sparsity of activity in the baseline networks let to a poor, centralized representation of the dataset akin to the notion of the gnostic (“grandmother”) cell in neuroscience [36]. The LC-SNN layer may be implicated in neuromorphic chips and used to operate on event-based data, a domain in which spiking neural networks are known to shine [37].

It remains to be seen whether methods of unsupervised learning with SNNs will be useful in learning representations of more complex datasets that, e.g., separate the categories of data for classification purposes. Whereas the optical character datasets considered in this work are relatively simple, to our knowledge, it has not yet been shown that SNNs trained with unsupervised, local learning rules can scale to the complexity of real-world image data. However, we have added a component to the SNN builder’s toolbox that may invigorate further lines of work on adapting SNNs to more complex machine learning problems. Using it, we have shown that local learning rules and adaptive neurons combined with competitive inhibitory interactions may be applied to different SNN architectural primitives with similar benefits for feature learning.

Acknowledgements

This work has been supported in part by Defense Advanced Research Project Agency Grant, DARPA/MTO HR0011-16-l-0006 and by National Science Foundation Grant NSF-CRCNS-DMS-13-11165.

7 References