First-spike based visual categorization using reward-modulated STDP

TL;DR

This paper introduces a reward-modulated STDP learning rule for spiking neural networks that enables efficient, online object recognition in natural images without external classifiers, demonstrating superior performance over traditional STDP.

Contribution

The novel use of reward-modulated STDP for training SNNs on visual categorization tasks without external classifiers is presented, enabling online learning and energy-efficient spike-based processing.

Findings

R-STDP outperforms classic STDP on multiple datasets

The approach enables online, adaptive learning in SNNs

Feature extraction and classification are achieved with at most one spike per neuron

Abstract

Reinforcement learning (RL) has recently regained popularity, with major achievements such as beating the European game of Go champion. Here, for the first time, we show that RL can be used efficiently to train a spiking neural network (SNN) to perform object recognition in natural images without using an external classifier. We used a feedforward convolutional SNN and a temporal coding scheme where the most strongly activated neurons fire first, while less activated ones fire later, or not at all. In the highest layers, each neuron was assigned to an object category, and it was assumed that the stimulus category was the category of the first neuron to fire. If this assumption was correct, the neuron was rewarded, i.e. spike-timing-dependent plasticity (STDP) was applied, which reinforced the neuron's selectivity. Otherwise, anti-STDP was applied, which encouraged the neuron to learn something else. As demonstrated on various image datasets (Caltech, ETH-80, and NORB), this reward modulated STDP (R-STDP) approach extracted particularly discriminative visual features, whereas classic unsupervised STDP extracts any feature that consistently repeats. As a result, R-STDP outperformed STDP on these datasets. Furthermore, R-STDP is suitable for online learning, and can adapt to drastic changes such as label permutations. Finally, it is worth mentioning that both feature extraction and classification were done with spikes, using at most one spike per neuron. Thus the network is hardware friendly and energy efficient.

Figures

a), we found that both R-STDP and STDP agree on the most confusing categories, that are cow, dog, and horse. However, thanks to the reinforcement learning, R-STDP not only decreased the confusion error, but also provided a more balanced error distribution.

Tables

Table 1. Table 1: Comparison of the network’s performance when using R-STDP and STDP.

Dataset	R-STDP	STDP						Shallow CNN
		First-Spike		Spike-Count		Max-Potential
		SVM	KNN	SVM	KNN	SVM	KNN
Caltech (Face/Motorbike)	$98.9$	$96.4$	$96.4$	$96.9$	$93.4$	$96.6$	$97.2$	$99.3$
ETH-80	$89.5$	$72.9$	$69.8$	$74$	$70.4$	$79.9$	$84.5$	$87.1$
NORB	$88.4$	$62.7$	$58.6$	$61.7$	$55.3$	$66$	$65.9$	$85.5$

Equations

$$\begin{array}{rl}{S}_1^l& =|I\otimes G({\theta }_l)|,\\ {\theta }_l& =\frac{(l-1)\times \pi }{4}+‌\frac{\pi }{8},\end{array}$$

$$t_{c1}^f(i)=\underset{j\in {\mathcal{P}}_{c1}(i)}{\min }\{t_{s1}^f(j)\},$$

$$v_i(t)=v_i(t-1)+\sum _{j\in {\mathcal{P}}_{s2}(i)}{W}_{ij}\times \delta \left(t-t_{c1}^f(j)\right),$$

$$t_{c2}^f(i)=\underset{j\in {\mathcal{P}}_{c2}(i)}{\min }\{t_{s2}^f(j)\},$$

$$\begin{array}{rl}\mathcal{F}& =\arg \underset{i}{\min }\{t_{c2}^f(i)|1\le i\le n\},\\ \mathcal{D}& =g\left(\mathcal{F}\right),\end{array}$$

$$\Delta W_{ij}\!=\!\begin{cases}a_{r}^{+}\!\times\!W_{ij}\!\times\!\left(1\!-\!W_{ij}\right)&\mbox{if}\ t^{f}_{c1}(j)-t^{f}_{s2}(i)\leq 0,\vspace{0.4cm}\\ a_{r}^{-}\!\times\!W_{ij}\!\times\!\left(1\!-\!W_{ij}\right)&\begin{aligned} \mbox{if}\ &t^{f}_{c1}(j)-t^{f}_{s2}(i)>0,\\ &\mbox{or the $j$th cell is silent,}\end{aligned}\end{cases}$$

$$\Delta W_{ij}\!=\!\begin{cases}a_{p}^{+}\!\times\!W_{ij}\!\times\!\left(1\!-\!W_{ij}\right)&\begin{aligned} \mbox{if}\ &t^{f}_{c1}(j)\!-\!t^{f}_{s2}(i)>0,\\ &\mbox{or the $j$th cell is silent,}\end{aligned}\vspace{0.4cm}\\ a_{p}^{-}\!\times\!W_{ij}\!\times\!\left(1\!-\!W_{ij}\right)&\mbox{if}\ t^{f}_{c1}(j)-t^{f}_{s2}(i)\leq 0,\end{cases}$$

$$W_{ij}=W_{ij}+\{\begin{array}{ll}\left(\frac{N_{miss}}{N}\right)\Delta W_{ij}& \text{mbox}ifarewardisreceived,\text{vspace}0.3cm\\ \left(\frac{N_{hit}}{N}\right)\Delta W_{ij}& \text{mbox}otherwise.\end{array}$$

$$g(i)=\lfloor (i-1)/k\rfloor +1.$$

Full text

Keywords: Spiking Neural Networks, Reinforcement Learning, Reward-Modulated STDP, Visual Object Recognition, Temporal Coding, First-Spike Based Categorization.

1 Introduction

Neurons in the brain are connected by synapses that can be strengthened or weakened over time. The neural mechanisms behind long-term synaptic plasticity, which is crucial for learning, have been under investigation for many years. Spike-timing-dependent plasticity (STDP) is an unsupervised form of synaptic plasticity, observed in different brain areas [1, 2, 3, 4], in particular in the visual cortex [5, 6, 7]. STDP works by considering the time difference between pre- and post-synaptic spikes. According to this rule, if the pre-synaptic neuron fires earlier (later) than the post-synaptic one, the synapse is strengthened (weakened). Studies have shown that STDP results in coincidence detectors, by which a neuron gets selective to a frequent input spike pattern leading to an action potential whenever the pattern is presented [8, 9, 10, 11]. STDP works well in finding statistically frequent features, however, as any unsupervised learning algorithm, it faces with difficulties in detecting rare but diagnostic features for important functionalities such as decision-making.

Several studies suggest that the brain’s reward system plays a vital role in decision-making and forming behaviors. This is also known as reinforcement learning (RL), by which the learner is encouraged to repeat rewarding behaviors and avoid those leading to punishments [12, 13, 14, 15, 16, 17, 18]. It is found that dopamine, as a neuromodulator, is one of the important chemical substances involved in the reward system [19], where its release is proportional to the expected future reward [17, 20, 21]. It is also shown that dopamine, as well as some other neuromodulators influences the synaptic plasticity, such as changing the polarity [22] or adjusting the time window of STDP [23, 24, 25, 26, 27].

One of the well-studied ideas to model the role of the reward system is to modulate or even reverse the weight change determined by STDP, which is called reward-modulated STDP (R-STDP) [28]. R-STDP stores the trace of synapses that are eligible for STDP and applies the modulated weight changes at the time of receiving a modulatory signal; a reward or punishment (negative reward).

In 2007, Izhikevich [29] proposed a R-STDP rule to solve the distal reward problem, where the reward is not immediately received. He solved the problem using a decaying eligibility trace, by which the recent activities are considered to be more important. He showed that his model can solve both classical and instrumental conditionings [30, 31]. In the same year, Farries and Fairhall [32] employed R-STDP to train neurons for generating particular spike patterns. They measured the difference between the output and target spike trains to compute the value of the reward. Also, Florian [33] showed that R-STDP is able to solve the XOR task by either rate or temporal input coding and learning a target firing rate. A year later, Legenstein et al. [34] investigated conditions, under which R-STDP achieves a desired learning effect. They demonstrated the advantages of R-STDP by theoretical analysis, as well as practical applications to biofeedbacks and a two-class isolated spoken digit recognition task. Vasilaki et al. [35] examined the idea of R-STDP on problems with continuous space. They showed that their model is able to solve the Morris water maze quite fast, while the standard policy gradient rule failed. Investigating capabilities of R-STDP continued by research from Frémaux et al. [36], in which conditions for a successful learning is theoretically discussed. They showed that a prediction of the expected reward is necessary for R-STDP to learn multiple tasks simultaneously. Studying the RL mechanism in the brain has gathered attentions in recent years, and researchers try to solve more practical tasks by reward-modulated synaptic plasticity [37, 38, 39].

Visual object recognition is a sophisticated task, at which humans are expert. This task requires both feature extraction, that is done by the brain’s visual cortex, and decision-making on the category of the object, for which higher brain areas are involved. Spiking neural networks (SNNs) have been widely used in computational object recognition models. In terms of network architecture, there are several models with shallow [40, 41, 42, 43], deep [44, 45, 46], recurrent [47], fully connected [48], and convolutional structures [40, 46, 49, 50]. Some use rate-based coding  [51, 52, 53], while others use the temporal coding [40, 43, 46, 48, 54]. Various kinds of learning techniques are also applied to SNNs, from backpropagation [49, 55], tempotron [43, 56], and other supervised techniques [52, 53, 57, 58], to unsupervised STDP and STDP-variants [42, 48, 59]. Although STDP-enabled networks provide a more biological plausible means of visual feature extraction, they need an external readout, e.g. support vector machines [46, 60], to classify input stimuli. Additionally, STDP tends to extract frequent features which are not necessarily suitable for the desired task. In this research, we present a hierarchical SNN equipped with R-STDP to solve the visual object recognition in natural images, without using any external classifier. Instead, we put class-specific neurons that are reinforced to fire as early as possible if their target stimulus is presented to the network. Thus, the input stimuli are classified solely based on the first-spike latencies in a fast and biologically plausible way. R-STDP enables our network to find task-specific diagnostic features, therefore, decreases the computational cost of the final recognition system.

Our network is based on Masquelier and Thorpe’s model [40] with four layers. The first layer of the network converts the input image into spike latencies based on the saliency of its oriented edges. This spike train goes under a local pooling operation in the second layer. The third layer of the network includes several grids of integrate-and-fire neurons that combine the received information of oriented edges and extract complex features. This is the only trainable layer in our network which employs R-STDP for synaptic plasticity. The signal (reward/punishment) for modulation of synaptic plasticity is provided by the fourth layer, in which the decision of the network is made. Our network only uses the earliest spike emitted by the neurons in the third layer to make a decision, without using any external classifier. If its decision is correct (incorrect) a global reward (punishment) signal is generated. Besides, in order to increase the computational efficiency, each cell in the network is allowed to spike only once per image. The motivation for at most one spike per neuron is not only computational efficiency, it is also biological realism [61, 62]. Decision-making without any classifiers with at most one spike per neuron, turns the proposed method into a well-suited candidate for the hardware implementation.

We performed two toy experiments to illustrate the abilities of R-STDP. We showed that the network employing R-STDP finds informative features using fewer computational resources than STDP. We also showed that R-STDP can change the behavior of a neuron, if needed, by encouraging it to unlearn what it has learned before. Thus, reusing a computational resource that is no longer useful. Moreover, we evaluated the proposed network on object recognition in natural images, using three different benchmarks, that are Caltech face/motorbike (two classes), ETH-80 (eight classes), and NORB (five classes). The results of the experiments demonstrate the advantage of employing R-STDP over STDP in finding task-specific discriminative features. Our network reached the performances (recognition accuracies) of $98.9\%$ on Caltech face/motorbike, $89.5\%$ on ETH-80, and $88.4\%$ on NORB datasets.

The rest of this paper is organized as follows: A precise description of the proposed network is provided in Section $2$. Then, in Section $3$, the results of the experiments are presented. Finally, in Section $4$, the proposed network is discussed from different points of view and the possible future works are highlighted.

2 Materials and Methods

In this section, we first describe the structure of the proposed network and the functionality of each layer. We then explain R-STDP, by which the neurons achieve reinforced selectivity to a specific group of input stimuli. Finally, we give a detailed description on the classification strategy that is used to evaluate the network’s performance.

2.1 Overall Structure

Similar to Masquelier and Thorpe’s model [40], our network consists of two simple and two complex layers, that are alternately arranged in a feed-forward manner (see Fig. 1).

The first layer of the network ($S1$) is a simple layer whose cells detect oriented edges in the input image. These cells emit a spike with a latency that is inversely proportional to the saliency of the edge. After $S1$, there is a complex layer ($C1$), which introduces some degrees of position invariance by applying local pooling operation. A $C1$ neuron propagates the earliest spike in its input window.

The second simple layer ($S2$) is made of integrate-and-fire (IF) neurons. A neuron in this layer, that detects a complex feature, receives its inputs from $C1$ neurons and generates a spike once its membrane potential reaches the threshold. For synaptic plasticity, we use a learning rule based on three factors: (1) pre-synaptic spike time, (2) post-synaptic spike time, and (3) a reward/punishment signal. This kind of synaptic plasticity provides the ability to control the behavior of the neurons in terms of their selectivity to input patterns.

The second complex layer ($C2$) of our network is the decision-making layer. Each neuron in this layer is assigned to a category and performs a global pooling operation over $S2$ neurons in a particular grid. Using a rank-order decoding scheme, the neuron which fires first, indicates the network’s decision about the input image. According to the decision made by the network, a reward/punishment signal is then generated, which drives in the synaptic plasticity of $S2$ neurons.

Implementation of the network is mainly done with C# and the code is available on ModelDB222https://senselab.med.yale.edu/ModelDB/.

2.2 Layer $S1$

The goal of this layer is to extract oriented edges from the gray scaled input image and turn them into spike latencies. To this end, the input image is convolved with Gabor filters of four different orientations. Thus, this layer includes four feature maps, each representing the saliency of edges in a particular preferred orientation.

Let $I$ be the grayscaled input image and $G(\theta)$ represent a Gabor filter (convolution kernel) with window size $5\times 5$, wavelength $2.5$, effective width $2$, and orientation $\theta$. Then, the $l$th feature map of layer $S1$ is generated using the following equations:

[TABLE]

where $\otimes$ is the convolution operator and $l\in\{1,2,3,4\}$. In order to introduce invariance to image negative operation, the absolute value of the convolution is used. Also, since vertical and horizontal edges are very common in natural images, a $\frac{\pi}{8}$ offset is applied to relax this bias [40].

For each of the feature maps (orientations), we put a 2D grid of the same size containing dummy neurons to propagate spikes. Using an intensity-to-latency encoding scheme, the obtained feature maps are converted to the spike latencies that are inversely proportional to the saliency of edges. In other words, the more salient the edge, the earlier the corresponding spike is propagated.

We implemented the proposed network in an event-based manner, where the spikes are sorted by their latencies in ascending order and propagated sequentially (i.e. the first spike is propagated in time step $t=1$, the second one in $t=2$, and so on).

2.3 Layer $C1$

Our first complex layer is a local pooling layer over the spikes coming from layer $S1$. Here, there are four 2D neuronal grids corresponding to each of the orientations. Each $C1$ neuron performs a local pooling operation over a window of size $\omega_{c1}\times\omega_{c1}$ and stride $r_{c1}$ (here we set $r_{c1}=\omega_{c1}-1$) on $S1$ neurons in a particular grid, after which, it emits a spike immediately after receiving its earliest input spike. This pooling operation decreases the redundancy of layer $S1$, and shrinks the number of required neurons, which consequently increases the computational efficiency. It also adds a local invariance to the position of oriented edges.

Let $\mathcal{P}_{c1}(i)$ be the set of all pre-synaptic neurons of the $i$th neuron in layer $C1$. Then, the firing time of this neuron is computed as follows:

[TABLE]

where $t_{s1}^{f}(j)$ denote the firing time of the $j$th neuron in $\mathcal{P}_{c1}(i)$.

Additionally, two kinds of lateral inhibition mechanisms are employed, which help the network to propagate more salient information. If a neuron located at position $(x,y)$ of the $i$th grid (orientation) fires, (1) the other neurons at the same position, but in other grids are prevented from firing, and (2) the latencies of the nearby neurons in the same grid are increased by a factor relative to their mutual Euclidean distance. In our experiments, inhibition is done for distances from $1$ to $5$ pixel(s) (floating-point distances are truncated to integer values) with inhibition factors $15\%,12\%,10\%,7\%,$ and $5\%$, respectively.

2.4 Layer $S2$

This layer combines the incoming information about oriented edges and turns them into meaningful complex features. Here, there are $n$ 2D grids of IF neurons with the threshold $\mathcal{T}$. Each neuron receives its inputs from a $\omega_{s2}\times\omega_{s2}\times 4$ window of $C1$ neurons through plastic synapses. A weight sharing mechanism is also applied to the neurons belonging to the same grid. This mechanism provides the ability of detecting a particular feature over the entire spatial positions. To be precise, let $\mathcal{P}_{s2}(i)$ be the set of all pre-synaptic neurons corresponding to the $i$th neuron. Then, the membrane potential of this neuron at time step $t$ is updated by the following equation:

[TABLE]

where $W_{ij}$ denotes the synaptic weight, $\delta$ is the Kronecker delta function, and $t^{f}_{c1}(j)$ is the firing time of the $j$th cell in layer $C1$. For each input image, a neuron in $S2$ fires if its membrane potential reaches the threshold $\mathcal{T}$. Also, these neurons have no leakage and are allowed to fire at most once while an image is being presented.

As the neurons fire, their synaptic weights - the feature they are detecting - are being updated based on the order of pre- and post-synaptic spikes, as well as a reward/punishment signal (see section reward-modulated STDP). This signal is derived from the activity of the next layer, that indicates the network’s decision. Besides, initial weights of the synapses are randomly generated, with mean $0.8$ and standard deviation $0.05$. Note that choosing small or midrange values for mean results in inactive, thus untrained, neurons. Moreover, large values for variance increase the impact of network’s initial state. Accordingly, high mean value with small variance is a suitable choice [46].

2.5 Layer $C2$

This layer contains exactly $n$ neurons, each is assigned to one of the $S2$ neuronal grids. A $C2$ neuron only propagates the first spike that is received from its corresponding neuronal grid. To put it differently, let $\mathcal{P}_{c2}(i)$ define the set of $S2$ neurons in the $i$th neuronal grid (for $i\in\{1,2,...,n\}$). Then, the firing time of the $i$th $C2$ neuron is computed as follows:

[TABLE]

where $t_{s2}^{f}(j)$ denote the firing time of the $j$th neuron in layer $S2$.

As mentioned before, the activity of $C2$ neurons indicates the decision of the network. To this end, we divide $C2$ neurons into several groups and assign each group to a particular category of input stimuli. Then, the network’s decision on the category of the input stimulus is assumed to be the one whose group propagates the earliest spike among other $C2$ groups.

Assume that there are $m$ distinct categories for the input stimuli, labeled from $1$ to $m$, and $n$ neuronal grids in layer $S2$. Accordingly, there are exactly $n$ neurons in layer $C2$, that are divided into $m$ groups. Let $g:\{1,2,...,n\}\mapsto\{1,2,...,m\}$ denote a function that returns the group’s index of a $C2$ neuron, and let $t^{f}_{c2}(i)$ denote the firing time of the $i$th neuron in layer $C2$. Then, the network’s decision $\mathcal{D}$ is made by

[TABLE]

where $\mathcal{F}$ is the index of a $C2$ neuron which fires first. The network receives reward (punishment) if its decision matches (does not match) the correct category of the input stimulus. If none of the $C2$ neurons fire, no reward/punishment signal is generated, thus, no weight-change is applied. Moreover, if more than one neuron fire early (with the minimum spike time), the one with the minimum index ($i$) is selected.

2.6 Reward-Modulated STDP (R-STDP)

We propose a reinforcement learning mechanism to update the pre-synaptic weights of $S2$ neurons. Here, the magnitude of weight change is modulated by a reward/punishment signal, which is received according to the correctness/incorrectness of the network’s decision. We also applied a one-winner-takes-all learning competition among the $S2$ neurons, by which the one with the earliest spike is the winner and the only one which updates its synaptic weights. Note that this neuron is the one determining the network’s decision.

To formulate our R-STDP learning rule, if a reward signal is received, then

[TABLE]

and in case of receiving a punishment signal, we have

[TABLE]

where $i$ and $j$ refer to the post- and pre-synaptic cells, respectively, $\Delta W_{ij}$ is the amount of weight change for the synapse connecting the two neurons, and $a_{r}^{+}$, $a_{r}^{-}$, $a_{p}^{+}$, and $a_{p}^{-}$ scale the magnitude of weight change. Furthermore, to specify the direction of weight change, we set $a^{+}_{r},a^{+}_{p}>0$ and $a^{-}_{r},a^{-}_{p}<0$. Here, our learning rule does not take into account the exact spike time difference and uses an infinite time window. According to this learning rule, the punishment signal reverses the polarity of STDP (a.k.a anti-STDP). In other words, it swaps long-term-depression (LTD) with long-term-potentiation (LTP), which is done to conduct the effect of aversion (avoid repeating a bad behavior), and $a^{+}_{p}$ is there to encourage the neuron to learn something else.

2.7 Overfitting Avoidance

In reinforcement learning problems, there is a chance of being trapped into local optima or overfitting to acquiring the maximum possible reward over the training examples. In order to help the network, exploring other possible solutions that are more general to cover both seen and unseen examples, we apply two additional mechanisms during the training phase. These techniques are only used for object recognition tasks.

2.7.1 Adaptive learning rate

Since the initial weights of the neurons are randomly set, the number of misclassified samples is relatively high at the beginning of the training phase (i.e. the performance is at the chance level). As training trials go on, the ratio of correctly classified samples to the misclassified ones increases. In the case of high rate of misclassification, the network receives more punishment signals, which rapidly weakens synaptic weights and generates dead or highly selective neurons that cover a small number of inputs. Similarly, when the rate of correct classification gets higher, the rate of reward acquisition increases as well. In this case, the network prefers to exclude misclassified samples by getting more and more selective to correct ones and remain silent for the others. In either cases, the overfitting happens due to the unbalanced impact of reward and punishment.

To tackle this problem, we multiply an adjustment factor to the amount of weight modification, by which the impact of correct and incorrect training samples is balanced over the trials. Assume that the network sees all of the training samples on each training iteration and let $N_{hit}$ and $N_{miss}$ denote the number of samples that are classified correctly and incorrectly in the last training iteration, respectively. If $N$ is the number of all training samples, then, the weight changes for the current training trial are modified as follows:

[TABLE]

Note that $N_{hit}+N_{miss}\leq N$, since there may be some samples for which none of the $S2$ neurons is active.

2.7.2 Dropout

In a reinforcement learning scenario, the goal of the learner is to maximize the expected value of reward acquisition. In our case, since the network only sees the training samples, it may find a few number of features that are sufficient to correctly classify almost all of the training samples. This issue appears to cause severe overfitting in face of complex problems and the network prefers to leave some of the neurons untrained. These neurons decrease the hit rate of the network over the testing samples, as they blindly fire for almost all of the stimuli.

Here, we employ the dropout technique [63], which causes a $C2$ neuron to be temporary turned off with the probability of $p_{drop}$. This technique gives rise to the overall involvement rate of the neurons, which in turn, not only increases the chance of finding more discriminative features, but also decreases the rate of blind firings (see Supplementary Materials: Dropout).

2.8 Classification

As mentioned before, the activity of the last layer, particularly the earliest spike in layer $C2$, is the only information that our network uses to make its final decision on the input stimuli. This way, we do not need external classifiers and increase the biological plausibility of the network at the same time.

To setup the network for a classification task with $m$ categories, we put $n=k\times m$ neuronal grids in layer $S2$, where $k$ is the number of features associated to each category. Then, we assign each $C2$ neurons to a category by the association function $g:\{1,2,...,n\}\mapsto\{1,2,...,m\}$ defined as follows:

[TABLE]

Then, the network uses equation (5) to classify the input stimuli. During the training phase, each network’s decision is compared to the label of stimulus and a reward (punishment) signal is generated, if the decision matches (mismatches) the label.

2.9 Comparison of R-STDP and STDP

In object recognition tasks, we make a comparison between our model, SNN with R-STDP, and the one that uses STDP. To this end, we first train the network using STDP and let the network extract features in an unsupervised manner. Next, we compute three kinds of feature vectors of length $n$ from layer $S2$:

• •

The first-spike vector. This is a binary vector, in which all the values are zeros, except the one corresponding to the neuronal grid with earliest spike.

• •

The spike-count vector. This vector saves the total number of spikes emitted by neurons in each grid.

• •

The potential vector. This vector contains the maximum membrane potential among the neurons in each grid, by ignoring the threshold.

After extracting feature vectors for both training and testing sets, K-nearest neighbors (KNN) and support vector machine (SVM) classifiers are used to evaluate the performance of the network. Moreover, the learning strategy and the STDP formula is the same as [40], and to make a fair comparison, we use the same values for parameters in both models. The only parameters that are explored for the STDP are the magnitudes of LTP and LTD.

3 Results

To evaluate the proposed network and learning strategy, we performed two types of experiments. First, we used a series of hand-made problems to show the superiority of R-STDP over STDP. Second, we assessed the proposed network on several object recognition benchmarks.

3.1 R-STDP Increases Computational Efficiency

Using STDP, when a neuron is exposed to input spike patterns, it tends to find the earliest repetitive sub-pattern by which the neuron reaches its threshold and fires [64, 65, 8, 11]. This tendency to favor early input spikes can be troublesome in case of distinguishing spike patterns that have temporal differences in their late sections.

Assume that there are several categories of input stimuli that possess the same spatial configuration (Fig. 2a). They also have identical early spikes. These patterns are repetitively presented to a group of IF neurons, for which the synaptic plasticity is governed by STDP and the one-winner-takes-all mechanism. If the neurons have low thresholds, one of them gets selective to the early common part of the input stimuli and inhibits the other neurons. Since the early parts are spatio-temporally the same among all of the input stimuli, there is no chance for the other neurons to fire and win the synaptic plasticity. Consequently, the overall activity of the neuronal group is the same for all of the input stimuli and classifies them into a single category.

As we will see below (Fig. 2c), there are also some STDP-based solutions for this problem, however they are inefficient in using computational resources. For example, if we increase the size of receptive fields along with the thresholds, neurons gain the opportunity to receive the last spikes as well as the early ones. Another possible solution is to use many neurons that locally inhibit each other and drop the one-winner-takes-all constraint. This way, regarding the initial random weights, there is a chance for the neurons to learn other parts of the input stimuli.

Here, we show that the R-STDP learning rule solves this issue in a more efficient way than STDP. For this purpose, we designed an experiment containing two $3\times 11$ input stimuli. The inputs are spatially similar, which means that spikes are propagated from similar locations of both inputs. As illustrated in Fig. 2a, each input is a 2D grid of white and gray squares. By white (gray) squares we denote locations, from which a spike is (is not) propagated. At the time of presenting any of these patterns to the network, spikes are propagated with a temporal order that is defined by the numbers written on the squares. According to this ordering, spikes with lower numbers are propagated earlier.

Since the input stimuli are artificial spike patterns, there was no need to apply Gabor filters, thus, they were fed directly into the layer $S2$. There, we put two neuronal grids with parameters $\omega_{s2}=3$ and $\mathcal{T}=3$. Therefore, each grid contained $1\times 9$ neurons to cover the entire input stimuli. We also set $a_{r}^{+}=0.05,a_{r}^{-}=-0.05,a_{p}^{+}=0.1$, and $a_{p}^{-}=-0.1$. The goal of the task was that the first (second) $C2$ neuron fires earlier for the first (second) pattern. We examined both STDP and R-STDP learning rules to see if the network finds discriminative features or not.

As shown in Fig. 2b, using STDP, the network extracted a non-discriminative feature, the shared one between both input stimuli. On the other side, the proposed reinforcement learning mechanism guided the neurons to extract features whose temporal order of appearance is the only thing leading to a successful pattern discrimination. We repeated this experiment for $100$ times using different random initial weights. Results showed that our network succeeded in $98\%$ of the times, while there were no chance for STDP to find the discriminative features. When we increased the threshold to $4$ (requiring at least two sub-patterns) and the size of the receptive fields to $11\times 11$ (covering the entire pattern), the network employing the STDP could also find discriminative features (see Fig. 2c) in 80% of the times.

3.2 Plastic Neurons

As mentioned earlier, the brain reward system plays an important role in the emergence of a particular behavior. In this part of the paper, we demonstrate the R-STDP’s capability of re-adjusting neurons’ behavior in an online manner.

We designed an experiment, in which the pre-defined desired behavior of the neurons is changed during the simulation. The experimental setup is very similar to the “Temporal Discrimination” task with similar input stimuli and parameter values, except that we swapped the target input stimuli during the training iterations (see Task 1 and Task 2 in Fig. 3a). As shown in Fig. 3b, at the beginning of the simulation, the desired behavior was that the neurons belonging to the first grid respond to the first stimulus earlier than those in the second grid, and vice versa. After $200$ iterations, when the convergence is fulfilled, we swapped the target stimuli. At this stage, since the neurons were exclusively sensitive to the previous target stimuli, they began to generate false alarms. Consequently, the network was receiving high rates of punishments for around $80$ iterations (see iterations $200$ to $280$ in Fig 3b), which in turn swapped LTD and LTP (see Materials and Methods: Reward-modulated STDP). As the network received punishments, the previously weakened (strengthened) synapses got stronger (weaker). Therefore, the sensitivity diminished for a while, and the neurons regained the possibility of learning something new. After iteration $300$, neurons found their new target stimulus and, once again, converged to the discriminative features (see the plots of synaptic weights in the top two rows in Fig. 3b).

In summary, R-STDP enables the neurons to unlearn what they have learned so far. This ability results in neurons with flexible behavior (plastic neurons), that are able to learn rewarding behavior in changing environments. This ability also helps the neurons to forget and escape from the local optima in order to learn something that earns more reward. Applying STDP in such a scenario does not work at all, since there is no difference between Task $1$ and Task $2$ from an unsupervised point of view.

3.3 Object Recognition

In this section, the performance of our network on categorization of natural images is evaluated. We begin with a description of the datasets that are used in our experiments. Then, we show how the network benefits from the reinforcement learning mechanism to extract features from natural images, followed by comparing R-STDP and STDP in object recognition tasks. Finally, we illustrate how the dropout and adaptive learning techniques reduce the chance of overfitting to the training samples.

3.3.1 Datasets

We used three well-known object recognition benchmarks to evaluate the performance of the proposed network. The first and easiest one is Caltech face/motorbike which is mainly used for demonstration purposes. The next two that are used to evaluate the proposed network are ETH-80 and small NORB. These datasets contain images of objects from different view points which make the task harder (see supplementary Fig. S1).

3.3.2 Reinforced Selectivity

The previous experiments showed that R-STDP enables the network to find informative and discriminative features, both spatially and temporally. Here, we show that R-STDP encourages the neurons to become selective to a particular category of natural images. To this end, we trained and examined the network on images from two categories of face and motorbike from the Caltech dataset.

In this experiment, we put $10$ neuronal grids for each category that were reinforced to win the first-spike competition in response to the images from their target categories. Therefore, the desired behavior of the network was that the neurons of the first $10$ grids get selective to the face category, while those in the other grids get selective to the motorbikes.

Fig. 4 illustrates the behavior of the network over the training iterations. Since the early iterations contained rapid changes, they are plotted wider. During early iterations, strong synaptic weights (see Materials and Methods: Layer $S2$) and $50\%$ dropout probability resulted in an unstable network whose neurons responded to random input stimuli. This chaotic behavior can be easily spotted on early iterations in the middle plot (see Fig. 4b). As the network continues training iterations, reward/punishment signals made neurons more and more selective to their target categories. As shown in Fig. 4b, after $200$ iterations, a quite robust selectivity appeared for the training samples, while on the testing samples, it is elongated for $300$ more iterations. This quick convergence on training samples is due to the fact that the network is relatively fast in finding features that successfully discriminate seen samples (see Fig. 4a). These primary features need to converge more to be applicable on testing samples, which requires even more iterations because of the adaptive learning rates. Moreover, we do not let the learning rate drops below $20\%$ of the values of parameters $a_{r}^{+}$, $a_{r}^{-}$, $a_{p}^{+}$, and $a_{p}^{-}$. This allows the network to continue convergence with a constant rate even if all of the training samples are correctly categorized (see Fig. 4c).

We repeated the experiment $30$ times with random initial weights and different training and testing samples and the performance achieved by the proposed network is $98.9\pm 0.4\%$ (mean $\pm$ std). When we tried a same network structure with STDP, $97.2\%$ was its best achievement (see Table 1).

3.3.3 Performance

We have shown how the proposed network successfully classified faces from motorbikes with high accuracy. Here, we examined the performance of the proposed network on the ETH-80 and NORB datasets that are more challenging (see Supplementary Materials: Datasets). The performance of the network is tested over the entire testing set after each training iteration, in which the network receives all of the training samples in random order.

For ETH-80 dataset, we configured the network to extract $10$ features per category, which resulted in $8\times 10=80$ features in total. The receptive field of each neuron in layer $S2$ was set in a way that it covered the whole input image. Here, nine instances of each category were presented to the network as the training samples, and the remaining were employed in the test phase. After performing $250$ training and testing iterations, the best testing performance of the network was reported.

Again, we repeated this experiment $30$ times, each time using a different training and testing set. As before, the network successfully extracted discriminative features (see Supplementary Fig. 2) and reached the performance of $89.5\pm 1.9\%$ (mean $\pm$ std). We also applied STDP to a network with the same structure. To examine the STDP performance, we used support vector machines with linear kernel and KNNs (K was changed from $1$ to $10$). According to the results, the accuracy achieved by this network is $84.5\%$, when the maximum potentials were used as the feature vectors and the classifier was KNN. Considering that the proposed network classifies input patterns solely based on the first-spike information, R-STDP definitely outperforms STDP. Table 1 provides the details of the comparison made between R-STDP and STDP.

By looking at confusion matrices (see Supplementary Fig. 3a), we found that both R-STDP and STDP agree on the most confusing categories, that are cow, dog, and horse. However, thanks to the reinforcement learning, R-STDP not only decreased the confusion error, but also provided a more balanced error distribution.

The same experiment was also performed on the NORB dataset. Again, we put $10$ neuronal grids for each of the five categories, whose neurons are able to see the entire incoming stimuli. The proposed network with R-STDP reached the performance of $88.4\pm 0.5\%$ (mean $\pm$ std) on testing samples, whereas STDP achieved $66\%$ at most. By reviewing confusion matrices of both methods, we found that both networks encountered difficulties mostly in distinguishing four-leg animals from humans, as well as cars from trucks (see Supplementary Fig. 3b). As before, R-STDP resulted in a more balanced error distribution.

Additionally, we compared the proposed network to convolutional neural networks (CNNs). Although the proposed network is not able to beat pre-trained deep CNNs such as VGG16 [66] (see Supplementary Materials: Comparison with Deep Convolutional Neural Networks), comparing it to a shallow CNN, with a similar network structure and same input would be a fair point. We repeated all of the object categorization experiments using a shallow CNN implemented with Keras neural networks API and Tensorflow as its backend. As shown in Table 1, the proposed network successfully outperformed the supervised CNN in both of the ETH-80 and NORB datasets.

3.3.4 Overfitting Problem

Overfitting is one of the most common issues in supervised or reinforcement learning scenarios. This problem got even worse by the emergence of deep learning algorithms. There are many studies focused on developing techniques that increase the generalization power of the learning algorithms. One of the mechanism that has shown promising empirical results on deep neural networks is the dropout technique [63]. This technique temporary reduces the complexity of the network by suppressing the activity of a specific number of neurons. This reduction in neuronal resources forces the network to generalize more in order to reduce the prediction error.

The proposed network is not an exception and has shown tendencies to overfit on the training samples through our examinations. Therefore, we adopted the dropout technique in our experiments. We also found that an steady learning rate does increase the chance of overfitting. Thus, we made use of dynamic learning rates with respect to the performance of the network (see Material and Methods: Overfitting Avoidance).

To show the impact of the aforementioned mechanisms, we repeated the object recognition experiments with different dropout probabilities and steady learning rates. Fig. 5 simultaneously shows the impact of both mentioned mechanisms on categorization of test samples. It is clear that when the adaptive learning rate mechanism is applied, the network achieved higher performances (solid lines). It is also shown that the dropout probability must be chosen according to the complexity of the dataset as well as the network. Since the NORB dataset contains more complex samples than the ETH-80, it tends more to overfitting on training samples. As a consequence, it needs more dropout rate to overcome this issue. The magnitude of this tendency is even clearer when the steady learning rates are used. To put it differently, faster convergence rate along with the complexity of the samples induce more overfitting, which in turn needs more dropout rate.

4 Discussion

Mammals are fast and accurate at visual object recognition. Their visual cortex processes the incoming data in a hierarchical manner, through which the complexity of neuronal preference is gradually increased. This hierarchical processing provides a robust and invariant object recognition [67, 68, 69, 70, 71]. Computational modeling of the mammalian visual cortex has been under investigation for many years. Developing a biologically plausible model not only enables scientists to examine their hypotheses with low cost, but also provides a human-like vision for artificially intelligent machines [40, 72, 73, 74, 75].

Deep convolutional neural networks (DCNNs) are the most successful works in this area [63, 76, 66, 77, 78]. The idea behind these networks is inspired by the hierarchical structure of the visual cortex. Despite the promising results obtain by DCNNs, they are not biologically plausible because of using supervised learning rules. In addition, they employ rate-based encoding scheme, which is both energy and resource consuming. There is another group of studies trying to use spiking neurons along with the unsupervised STDP learning rule [40, 42, 46, 48, 59]. These models are more biologically plausible, but they cannot beat DCNNs in terms of accuracy. In theory, spiking neural networks (SNNs) have more computational power than DCNNs, however, they are harder to control because of the complex dynamics and high dimensional space of effective parameter. Furthermore, since most of them are trained in an unsupervised manner, the classification step is done by an external classifier or statistical methods.

Here, we solved the object recognition task using a hierarchical SNN equipped with a reinforcement learning rule called R-STDP [28]. There are several studies showing that the brain uses RL to solve the problem of decision-making [15, 16, 17, 18]. Therefore, it is a suitable choice for training class-specific neurons that are able to decide on the class of the input image. Therefore, we put one step further developing a more biologically plausible model which is able to perform the visual categorization totally on its own. The proposed network functions in the temporal domain, where the information is encoded by spike times. The input image is first convolved with oriented Gabor filters and a spike train is generated based on a latency-to-intensity coding scheme. The resulting spikes are then propagated toward the feature extraction layer. Using R-STDP, the proposed network successfully found task-specific diagnostic features using neurons that were pre-assigned to the class labels. In other words, each neuron was assigned to a class a priori, where its desired behavior was to respond early for the instances belonging to the specified class. To decrease the computational cost even more, neurons were forced to fire at most once for an input image and the latency of their spike is considered as the measure of stimulus preference. Therefore, if a neuron fired earlier than the others, it would have received its preferred stimulus. This measure of preference served as an indicator for the network’s decision. That is to say, when a neuron belonging to a particular class fired earlier, the network’s decision was considered to be that class.

Through our experiments, we compared R-STDP to STDP from different aspects. We showed that R-STDP can save computational resources. This was clarified by a hand-designed discrimination task, in which the order of spikes was the only discriminative feature. R-STDP solved the problem using minimal number of neurons, synapses, and threshold, whereas STDP needed more neurons, more synapses, and higher thresholds. This drawback for STDP is due to the fact that it tends to find statistically frequent features [8, 9, 10, 11], which are not necessarily the diagnostic ones. As a consequence, one needs to use either more neurons or more synapses to ensure that the diagnostic features will be eventually found. On the other hand, since R-STDP informs the neurons about their outcomes, they can function better using minimal resources.

After having demonstrated the advantages of R-STDP in finding diagnostic features, we investigated how well it can be combined with a hierarchical SNN for solving both visual feature extraction and object categorization in a biologically plausible manner. We evaluated the proposed network and a similar network which uses STDP, as well as a CNN with the same structure, on three datasets of natural images Caltech Face/Motorbike, ETH-80 and NORB. The last two contain images of objects from different viewpoints, which made the task harder. When we compared the performances obtained by the networks, we found that R-STDP strongly outperforms STDP and the CNN with same structure. An even more interesting point is that the proposed network achieved this superiority decisions solely based on the first-spikes, while in the case of the others, even the powerful classifiers like SVMs and error back-propagation were not of any help.

To compare R-STDP with STDP, both networks used the same values for parameters except the learning rate (see Materials and Methods: Comparison of the R-STDP and STDP). However, one can use STDP with higher number of neurons and tuned thresholds to compensate the blind unsupervised feature extraction and achieve better performances [60]. Again, we conclude that R-STDP helps the network to act more efficiently in consuming computational resources.

Putting everything together, the proposed network has the following prominent features:

• •

Robust object recognition in natural images.

• •

Each neuron is allowed to spike only once per image. This results in a huge reduction of energy consumption.

• •

Decision-making (classification) is performed using the first-spike latencies instead of powerful classifiers. Therefore, the biological plausibility of the model is increased.

• •

Synaptic plasticity is governed by RL (the R-STDP rule), for which supporting biological evidence can be found [28], and which allows to extract highly diagnostic features.

Our network can be interesting for neuromorphic engineering [79], since it is both biologically plausible and hardware-friendly. Although hardware implementation and efficiency is out of the scope of the current paper, we believe that the proposed network can be implemented in hardware in an energy-efficient manner for several reasons. Firstly, SNNs are more hardware friendly than classic artificial neural networks, because the energy-consuming “multiply-accumulator” units can be replaced by more energy-efficient “accumulator” units. For this reason, studies on training deep convolutional SNNs (DCSNNs) [44, 46] and converting DCNNs into DCSNNs [80], as well as restricted DCNNs [81, 82, 83] have gained interests in recent years. Secondly, most SNN hardwares use event-driven approaches by considering spikes as events. This way, energy consumption increases with the number of spikes. Thus, by allowing at most one spike per neuron, the proposed model is as efficient as possible. Finally, the proposed learning rule is more suitable for online, on-chip learning than error backpropagation in deep networks, where updating weights based on high-precision gradients brings difficulties for hardware implementation.

To date, we could not find any other works possessing the aforementioned features. To mention one of the closest attempts, Gardner et al. [84] tried to classify Poisson-distributed spike trains by a readout neuron equipped with R-STDP. Although their method is working, it cannot be applied on natural images as it is, because of their time-based encoding and target labeling. There is another related work by Huerta and Nowotny [85]. In this work, the authors designed a model of the RL mechanism which occurs in the mushroom body. They applied their RL mechanism on a pool of randomly connected neurons with $10$ readout neurons to classify handwritten digits. Our work is different from theirs in several aspects. First, we used a hierarchical structure based on the mammalian visual cortex, while they used randomly connected neurons. Second, we used the R-STDP learning rule, whereas they employed a probabilistic approach for the synaptic plasticity. Third, the input of our network were natural images using intensity-to-latency encoding, while they used binary encoding with a threshold on artificial images.

Although the results of the proposed network were significantly better than the network employing STDP with external classifiers, they are still not competitive to the state-of-the-art deep learning approaches. One of the limitations to the current method is using only one trainable layer. Besides, the receptive field of the neurons in the last layer are set to be large enough to cover an informative portion of the image. As a result, the network cannot resist high rates of variations in the object, unless using more and more number of neurons. Extending the number of layers in the current network is one of the directions for future research. Going deeper seems to improve the performance by providing a gradual simple to complex feature extraction. However, deeper structure needs more parameter tuning, and a suitable multi-layer synaptic plasticity rule. Recent studies have also shown that combining deep networks and RL can lead to outstanding results [86, 87].

Another direction for the future research is to use the RL for learning semantic associations. For example, STDP is able to extract features for different kinds of animals in different viewpoints, but it is not able of relating all of them into the category of “animal”, because different animals have no reason to co-occur. Or, it can extract features for the frontal and profile face, but it cannot generate an association putting both in the general category of “face”. On the other hand, by a reinforcement signal and using learning rules like R-STDP, neurons are not only able to extract diagnostic features, but also learn relative connections between categories and create super-categories.

Acknowledgment

The authors would like to thank Dr. Jean-Pierre Jaffrézou for proofreading this manuscript, as well as Amirreza Yousefzadeh and Dr. Bernabé Linares-Barranco for providing useful hardware-related information.

This research received funding from the European Research Council under the European Union’s 7th Framework Program (FP/2007–2013)/ERC Grant Agreement no. 323711 (M4 project), Iran National Science Foundation: INSF (No. 96005286), and Institute for Research in Fundamental Sciences (BS-1396-02-02), Tehran, Iran.