Robustness from structure: Inference with hierarchical spiking networks on analog neuromorphic hardware

TL;DR

This paper investigates how hierarchical spiking neural networks can maintain robust probabilistic inference when implemented on analog neuromorphic hardware, despite physical distortions and imperfections.

Contribution

It demonstrates that hierarchical leaky integrate-and-fire networks are inherently robust to hardware-induced distortions through simulations and neuromorphic emulation.

Findings

Hierarchical networks show robustness to physical distortions.

Analog neuromorphic hardware can effectively implement probabilistic inference.

Hierarchical structure enhances robustness compared to flat networks.

Abstract

How spiking networks are able to perform probabilistic inference is an intriguing question, not only for understanding information processing in the brain, but also for transferring these computational principles to neuromorphic silicon circuits. A number of computationally powerful spiking network models have been proposed, but most of them have only been tested, under ideal conditions, in software simulations. Any implementation in an analog, physical system, be it in vivo or in silico, will generally lead to distorted dynamics due to the physical properties of the underlying substrate. In this paper, we discuss several such distortive effects that are difficult or impossible to remove by classical calibration routines or parameter training. We then argue that hierarchical networks of leaky integrate-and-fire neurons can offer the required robustness for physical implementation and demonstrate this with both software simulations and emulation on an accelerated analog neuromorphic device.

Equations

$$z_{k}^{(t)}=\left\{\begin{array}[]{ll}1&\quad\text{if $t_{s}<t<t_{s}+{\tau_{\mathrm{ref}}}$}\quad,\\ 0&\quad\text{otherwise}\quad.\end{array}\right.$$

$$p(z_k=1|{\mathbf{z}}_{\setminus k})=\sigma (u_k^{\mathrm{abstr}})=\frac{1}{1+e^{-u_k^{\mathrm{abstr}}}}.$$

$$u_k^{\mathrm{abstr}}({\mathbf{z}}_{\setminus k})=\sum _{j=1}^n{W}_{kj}{z}_j+b_k.$$

$$p(\mathbf{z})=\frac{1}{Z}\exp [-E(\mathbf{z})]=\frac{1}{Z}\exp \left[\frac{1}{2}{\mathbf{z}}^T\mathbf{W}\mathbf{z}+{\mathbf{z}}^T\mathbf{b}\right],$$

$$C_{\mathrm{m}}\frac{d}{dt}{u}_k=g_{\mathrm{l}}(E_{\mathrm{l}}-u_k)+I_k^{\mathrm{syn}}+I_k^{\mathrm{noise}}+I_k^{\mathrm{ext}}.$$

$$p(z_k=1|{\mathbf{z}}_{\setminus k})=\sigma \left(\frac{{\bar{u}}_k-{\bar{u}}_k^0}{\alpha }\right),$$

$$b_k$$

$$W_{kj}$$

$$\times \left[\frac{1-e}{e}-\frac{{\tau }_{\mathrm{eff}}}{{\tau }^{\mathrm{syn}}}\left(e^{-\frac{{\tau }^{\mathrm{syn}}}{{\tau }_{\mathrm{eff}}}}-1\right)\right],$$

$$\Delta w_{ij}$$

$$\Delta b_i$$

$$R^{\mathrm{train}}$$

$$R^{\mathrm{test}}$$

$$R^{\mathrm{train}}$$

$$R^{\mathrm{test}}$$

$${\tilde{b}}_{ki}=p(v_i=1)\cdot w_{ki},$$

$${\tilde{b}}_k=\sum _i{\tilde{b}}_{ki}+b_k,$$

$${\tilde{b}}_k=\sum _{i=1}^{144}{w}_{ki}{v}_i.$$

$$R^{\mathrm{train}}$$

$$R^{\mathrm{test}}$$

Full text

I Introduction

Over the past decades, research in neural networks has undergone an interesting branching process. On the one hand, the machine learning community has gradually increased its interest in what were originally brain-inspired neural networks. These efforts have been crowned by impressive recent success [1, 2], which has, however been obtained at the price of having strayed away from biologically plausible dynamics. On the other hand, modern computational neuroscience is pushing for ever more complex and biologically realistic simulations, in the hope to uncover the biological details of information processing in the brain [3]. Today, these two communities are investigating network models that have little in common with each other.

In the meantime, the neuromorphic community has to master an increasingly difficult balancing act. At its core, the neuromorphic approach aims to mimic various features of the neocortex in silico. For example, an essentially ubiquitous feature of neuromorphic devices is that they are built to emulate spiking neurons [4, 5, 6, 7, 8, 9]. However, one core argument for building these devices is the hope to use them to unlock the brain’s computational power by moving beyond the von Neumann computing paradigms. Consequently, a driving question for the neuromorphic community might be formulated as follows: is it possible to find relevant applications for spiking neural networks that can then profit from the typical advantages of a physical implementation such as inherent parallelism, high speed and low power consumption? The findings discussed in this article suggest a promising path towards finding an answer.

This issue is even more pronounced in the case of analog hardware, since it imposes additional constraints that stem from the physics of systems themselves. As opposed to digital systems, be they von Neumann or neuromorphic, which have the benefit of essentially perfect precision and control, analog systems have to deal with inherent imperfections. These imperfections concern, on the one hand, the equations of motion of the network components, which must obey the physics of the substrate and can therefore only provide an approximation of the target dynamics. On the other hand, the degree of precision to which the parameters of these equations can be tuned certainly depends on the hardware design, but is always fundamentally limited by fixed-pattern variations and temporal noise. Imperfections in the network dynamics and parameters necessarily distort the behavior of the emulated networks, which usually impairs their performance to some degree [10].

The question of parameter control (i.e., calibration, post-calibration tuning and training) is an essential one. It constitutes a perennial challenge for analog neuromorphic system design and operation, and has therefore been often addressed in literature [11, 12, 13, 10]. A thorough discussion of parameter calibration and in-the-loop training of analog circuits can be found in [14], which represents a complement of the present study. In the present manuscript, we are mainly concerned with the distortions to the network dynamics that are imposed by the physics of the emulation device and that cannot be directly addressed by, e.g., calibration.

We begin by identifying a Bayesian spiking network model with valuable computational properties (Sec. II). It is able to learn a probabilistic model of input data and can subsequently be used as both a generative and a discriminative model — a feature that is difficult to achieve even with abstract neural networks [15]. Here, we focus on its discriminative properties. This model is, in general, susceptible to hardware-induced distortions, as we discuss in detail in Sec. III. As a particular example, we characterize these effects on the Spikey chip [5] — the neuromorphic system that we use as an emulation back-end. Despite our model’s ostensible lack of robustness, we argue that, when endowed with a particular, hierarchical connectivity structure, it becomes robust to the studied hardware-induced distortions. We substantiate this conjecture with both software simulations (Sec. IV) and hardware emulations (Sec. V). In particular, without any training to compensate for parameter noise on the hardware, we show that the network only loses a relatively small fraction of its initial performance when running on Spikey. This represents, to our knowledge, the first scalable implementation of a hierarchical probabilistic network in accelerated analog neuromorphic hardware.

II LIF-based Boltzmann machines

Continued technological advances in large-scale processing (parallel CPU and GPU architectures) have enabled the recent resurgence of artificial neural networks. Already envisioned for decades as theoretical models of brain-like architectures [16, 17], neural networks now routinely outperform their rival models at pattern recognition tasks [1]. Here, we focus on one particular neural network model — a spiking variant of the Boltzmann machine (BM) [18], which has been shown to be compatible with biologically plausible and hardware-implementable spiking neurons [19]. We now briefly describe the structure and dynamics of these stochastic networks of leaky integrate-and-fire (LIF) neurons and discuss potential problems that can arise from their implementation in analog hardware.

In the neural sampling framework [20], a population of $n$ neurons represents a binary random vector $\boldsymbol{z}\in\{0,1\}^{n}$. The refractory state of a neuron following a spike at time $t_{s}$ is chosen to represent the 1-state of the associated random variable (see also Fig. 1A):

[TABLE]

In the abstract model of neural sampling, the probability of each neuron to be in the 1-state is given by a logistic activation function

[TABLE]

Such an abstract neuron’s membrane potential $u^{\mathrm{abstr}}_{k}$ has a resting-state value of $b_{k}$ and is linearly influenced by the state $\boldsymbol{z}_{{\mathrm{\setminus}}k}$ of all other neurons in the network via (symmetric) synaptic weights $W_{kj}=W_{jk}$:

[TABLE]

It can then be shown that, under these assumptions, a network of such stochastic neurons will sample from a Boltzmann distribution

[TABLE]

with the partition function $Z$ as a normalization factor.

In order to achieve similar dynamics with LIF neurons, an equivalent firing regime needs to be established. In the LIF sampling framework [19], each neuron receives two kinds of spiking input: information-encoding input from other neurons in the network and diffuse background input that represents the source of stochasticity, modeled by Poisson sources. These input spike trains generate two types of current onto the membrane, which we denote by $I^{\mathrm{syn}}$ and $I^{\mathrm{noise}}$, respectively:

[TABLE]

Here, ${C_{\mathrm{m}}}$ is the membrane capacitance, ${g_{\mathrm{l}}}$ and ${E_{\mathrm{l}}}$ are the leak conductance and potential, and $I^{\mathrm{ext}}_{k}$ is an external current that determines the bias $b_{k}$. While in general noisy LIF neurons do not have a logistic activation function, as required in Eqn. 2, it has been shown that in a high-conductance state the LIF activation function can be well approximated by a logistic function that is scaled with parameters $\alpha$ and $\bar{u}_{k}^{0}$ [19, 21]:

[TABLE]

where $\bar{u}_{k}=\left\langle u_{k}\right\rangle_{t}$ represents the noise-free membrane potential of the $k$th neuron. This equivalence to the abstract model enables an LIF neuron to sample correctly from its conditional distribution $p(z_{k}|z_{{\mathrm{\setminus}}k})$. The translation of the Boltzmann parameters $(\boldsymbol{W},\boldsymbol{b})$ in Eqn. 4 to the conductance-based LIF domain (synaptic weights $\boldsymbol{w}$, bias potentials $\boldsymbol{\bar{u}}^{0}$) can then be achieved using the following rules:

[TABLE]

where $\mu$ is the mean of the free membrane potential, ${\tau_{\mathrm{eff}}}={C_{\mathrm{m}}}/({g_{\mathrm{l}}}+g^{\mathrm{syn}})$ is the effective membrane time constant in the high-conductance state, $g^{\mathrm{syn}}$ the total synaptic conductance, $\tau^{\mathrm{syn}}$ the synaptic time constant and $E^{\mathrm{rev}}$ the synaptic reversal potential. This allows a direct mapping of abstract BMs to networks of LIF neurons that sample accurately from their target distribution (Fig. 1B-D).

It is important to note that such networks are not merely more complicated replicas of classical machine learning approaches. In addition to being able to emulate the computational power of traditional Boltzmann machines, these spiking networks can also harness certain biological mechanisms to extend their functionality. It has, for example, been shown, that when endowed with short-term synaptic plasticity, LIF-based BMs can become good generative models of their learned datasets, while at the same time maintaining a high classification performance when presented with individual data samples [15].

Such LIF networks are now amenable to training with any of the established algorithms for BMs. While backpropagation [22] is more difficult to implement in a biologically plausible network, other methods exist that are more compatible with Hebbian learning. The wake-sleep algorithm, in particular, requires each synapse to only have access to the activity of its pre- and its postsynaptic neuron [23]:

[TABLE]

This learning rule tries to adapt the activity $\boldsymbol{z}_{\mathrm{model}}$ of the network in the “dreaming” phase, during which it evolves freely, to its activity $\boldsymbol{z}_{\mathrm{data}}$ in the “awake” state, where it is constrained by data, i.e., where some of the units are clamped to particular values. Despite its simplicity, this learning algorithm can be used to achieve high classification rates on various machine learning datasets [24].

III Critical distortions in physical implementations

The distribution that an LIF network samples from is uniquely determined by the neuro-synaptic dynamics and parameters. Any deviation from the model specification will alter the sampled distribution and, in general, restrict the network’s ability to perform correct inference in the learned sample space. Mapping this model to an imperfect physical substrate is therefore not straightforward. In this article, we study three types of distortions of network dynamics that are caused by mapping to an analog silicon substrate.

First, we consider spike transmission delays. Since we are using point neurons, we can describe all delays as being synaptic delays. Many analog neuromorphic devices, including the one we use later on, are mixed-signal systems, meaning that spikes are transmitted digitally. Consequently, digitization, transport of the digital data, and the conversion back to the analog domain in the synapses contribute to synaptic delays. While these delays may be short in terms of wall-clock time, they become particularly critical in accelerated systems. In such systems, the neuronal and synaptic dynamics that define the characteristic time scale on which the network evolves can be orders of magnitude smaller than in biology, potentially entering the range of synaptic transmission delays [10]. Regardless of the exact nature of a network performing neural sampling, in order for each neuron to be able to calculate its correct conditional distribution $p(z_{k}|\boldsymbol{z}_{{\mathrm{\setminus}}k})$, the information gathered by a neuron from its incoming PSPs must coincide with the true state of the corresponding presynaptic neurons, as required by Eqn. 3. This temporal coincidence is disrupted by delays, which thereby distort the sampled distribution, as exemplified in Fig. 1E.

Second, while most neuromorphic systems have controllable neuron and synapse parameters, these can only be configured within a certain range and resolution. As an example, consider the membrane time constant ${\tau_{\mathrm{m}}}$ of a neuron. This time constant can be considered to define the reaction speed of a neuron to external stimuli. In neuromorphic systems, ${\tau_{\mathrm{m}}}$ is usually configured with an adjustable leak conductance, which can not become arbitrarily large. Such a limit in the reaction speed of neurons can impair the functionality of the entire network. For LIF neurons, a large ${\tau_{\mathrm{m}}}$ slows the saturation of the activation function (Fig. 1F) and thereby distorts the logistic shape (Eqn. 6) required for sampling from Boltzmann distributions.

Third, temporal noise can also affect computation. Depending on the particular in-silico implementation, any analog system will be subject to some degree of temporal noise on all of its electronic signals, including those that directly influence neuro-synaptic dynamics. In our particular case, the largest temporal noise component affects refractory times ${\tau_{\mathrm{ref}}}$. Since the relevant neuron and synaptic circuits are physically disconnected on the chip, the spike-to-spike variation of ${\tau_{\mathrm{ref}}}$ is independent from the synaptic time constant, which can be considered as fixed. Consequently, the state $z_{k}(t)$ of a neuron will not coincide anymore with the information it transmits via PSPs to its postsynaptic partners (Fig. 1G), leading to a distortion of the sampled probability distribution in a conceptually similar manner as synaptic delays do.

In this study, we use the Spikey single-chip system as a physical emulation substrate [5]. This mixed-signal device combines analog components for modeling membrane and synapse dynamics with digital circuitry for the spike-based communication. Fig. 2 shows a photo of the device, along with a sketch of the neuron circuit which illustrates the origin of the three distortive effects discussed above.

The overlay in Fig. 2A shows how a spike emitted by a neuron (blue triangle) travels through communication buses (white line) to a synapse driver (red pentagon), which generates a voltage ramp that is fed into the synapse array (green circles). Inside the synapse, the voltage ramp is converted to an approximately exponential signal that is added to the total synaptic conductance of the neuron circuit. This sequence of processing stages causes the effective synaptic delays seen in Fig. 2B.

The neuron schematic in Fig. 2A explains the cause of non-logistic activation functions and noisy refractory times.

All reversal potentials are connected to the membrane by conductances that saturate at a certain amplitude. The maximum total conductance defines a minimum achievable effective membrane time constant, which limits the gain of the LIF activation function, as seen in Fig. 2C.

The duration of the refractory time is determined by a monoflop controlled by a current ${I_{\mathrm{ref}}}\propto 1/{\tau_{\mathrm{ref}}}$. In order to offset, at least to some extent, the effect of delays (see Fig. 1E) we require long refractory times, i.e., small currents, which cause some of the transistors in the monoflop to leave saturation and operate in a sub-threshold regime. This transition is accompanied by an increase in the relative amplitude of temporal noise, which increases the variability of ${\tau_{\mathrm{ref}}}$. This represents the primary cause of the large spike-to-spike fluctuations of the refractory time seen in Fig. 2D.

Having identified the origins for critical distortions in physical implementations of LIF sampling, we next turn to the central question of this study: is it possible to recover the computational capabilities of LIF sampling networks by finding a network architecture that is robust to these substrate-induced effects?

IV Robust hierarchical LIF networks

The general framework of LIF sampling does not impose any restrictions on the network topology apart from the requirement of a zero-diagonal symmetric synaptic weight matrix $W_{ii}=0,\;\boldsymbol{W}=\boldsymbol{W}^{T}$. However, imposing further restrictions on connectivity is of practical use.

When building a network that is able to learn and generalize from data, a rather natural hierarchization consists in subdividing the network into a layer representing the “visible” data, one or more hidden layers that recognize common features of data samples, and a final classification layer that assigns each sample a particular category or label. Indeed, this has been the guiding principle behind hierarchical neural networks, from multilayer perceptrons to deep convolutional nets [25]. In the case of BMs, the further removal of lateral connections within a layer has proven particularly beneficial for learning [26]. The resulting networks are so-called restricted Boltzmann machines (RBMs) and can be emulated by LIF networks with appropriate parameters as described in Sec. II.

The core insight of our present work is that hierarchical LIF networks that emulate RBMs exhibit notable robustness to the hardware-induced distortions discussed above. In this section, we argue why this is the case and demonstrate this robustness with software simulations of such a network with 3 layers (Fig. 3A).

Synaptic delays and noisy refractory times have similar effects on the sampled distribution. However, the nature of the information flow in LIF-based RBMs is expected to counter them both simultaneously. When presented with unambiguous input data, the mean firing rates of the visible neurons $v_{k}$ are fixed; in our application, for example, they encode the grayscale values $g_{k}\in\{0,1,\dots,255\}$ of pixels in the input image: $p(v_{k}=1)=g_{k}/255$. In this regime, spike transmission delays have no effect, as the visible layer essentially operates in a rate-based mode for which time shifts do not matter. In this operating mode, the refractory noise is also averaged out.

Transmission delays and noisy refractory times remain critical for the interaction between hidden and label neurons. However, this interaction is comparatively weak in our 3-layer architecture (see Fig. 3A). In real-world scenarios, the label space typically has a much smaller dimensionality than the input space. Each hidden neuron therefore receives input from many visible neurons but only from few label neurons. Therefore, even though the visible-to-hidden synaptic weights are approximately as large as those between the hidden and label layer, the summed input from the visible layer is completely dominant by virtue of sheer numbers. Therefore, as the hidden layer is mostly driven by the input layer, the distorted interaction between the hidden and label neurons is likely to become insignificant.

The finite membrane time constant, on the other hand, can affect neurons in all layers and can not be neglected. However, this effect can be countered, at least to some extent, by the nature of the sampled distribution in well-trained networks. Wake-sleep training has the effect of carving of deep troughs in the network’s energy landscape $E(\boldsymbol{z})$. These energy minima (probability maxima) correspond to particular patterns in each of the network layers, which are local attractors in the state space. Thus, if the deviations in the sampled distribution are small, the attractor landscape will not change significantly. Consequently, when the visible layer is clamped to input data, the above layers are still likely to fall into the corresponding attractor state, thus conserving the classification performance.

We tested these predictions in a series of software simulations. We trained a 3-layer LIF-based RBM (Fig. 3A) on a reduced version of the MNIST dataset [27]. To ensure compatibility with the Spikey chip, the network size was restricted to 144 visible, 50 hidden and 6 label neurons. The $12\times 12$ pixel images belonging to 6 digit classes (“0”, “1”, “2”, “3”, “4”, “7”) were produced by first reducing the MNIST digit resolution, followed by binarization of the pixel values (Fig. 3B). For each class, both the training and the test set consisted of 20 randomly chosen images. The training consisted only of layer-by-layer pre-training with a wake-sleep-style algorithm [28]. We have deliberately refrained from fine-tuning the weights with, e.g., backpropagation, in order to maintain compatibility with Hebbian plasticity.

Fig. 3C-E show the simulated effects of the three hardware-induced distortion mechanisms discussed above. As expected, neither synaptic transmission delays (Fig. 3C) nor variability of refractory times (Fig. 3D) affected the performance of the network significantly. Over a surprisingly large range of membrane time constants, the classification rate remained almost unaffected. Only after the activation function became significantly distorted by large ${\tau_{\mathrm{m}}}$ did the attractor landscape change significantly enough to cause a decay in the classification rate (Fig. 3E). Overall, within the parameter ranges of the Spikey chip, our network remained only weakly affected by the studied mechanisms.

However, since in a later step the network was mapped to the hardware without any further training, we needed to also consider the effect of discretized synaptic weights. By default, synaptic weights on the Spikey chip are only controllable up to 4-bit precision [5]. It is important to note that this does not pose a fundamental problem to networks of this type; the effects of weight discretization can be countered by appropriate in-the-loop training, as discussed in, e.g., [29, 14]. Here, we only take this effect into account as a preparation of the hardware experiments in Sec. V. Fig. 3F shows the effect of weight discretization on the network’s classification performance. For the Spikey chip, the performance decay lies at approximately $5.6\text{\,}\frac{\mathrm{\char 37\relax}}{}$. Note that this effect is significantly larger than the effects caused by each of the other distortion mechanisms.

A combined simulation of all distortive effects was used to provide a reference for the later emulation on Spikey. All effects were simulated with amplitudes corresponding to values measured on Spikey (blue bars in Fig. 3C-F, see also Fig. 2). In the ideal, undistorted case, the LIF network had a classification performance of

[TABLE]

which was reduced to

[TABLE]

when all distortive effects were simultaneously present. A comparison to Fig. 3F shows that most of this performance decay was due to the 4-bit weight discretization.

V Neuromorphic implementation

The mapping of the network to Spikey required a series of modifications, which we discuss in the following.

In the previous section, we argued that visible neurons essentially operate in a rate-based mode during clamping. This allows the activity $\nu_{i}$ of each visible neuron to be modeled as an effective bias

[TABLE]

to the $k$th hidden neuron, where $v_{i}$ represents the state of the $i$th visible neuron and $w_{ki}$ the synaptic weight between the $i$th visible and the $k$th hidden neuron. The complete visible layer can then be omitted altogether and replaced by an effective bias $\tilde{b}_{k}$ for each hidden neuron:

[TABLE]

where $b_{k}$ is the original bias of the $k$th hidden neuron. In our particular case, since we use $12\times 12$–pixel binarized images and have set all biases to zero during training in order to simplify the transition to hardware, this reduces to

[TABLE]

On the chip, biases are implemented as high-frequency regular spike trains connected to the hidden neurons with weights $w^{b}_{k}$. For an arbitrary synaptic kernel scaled with $w^{b}_{k}$, the average effect of a regular spike train on the membrane potential of an LIF neuron is proportional to $w^{b}_{k}$. Therefore, $\tilde{b}_{k}$ can be controlled, within the imposed 4-bit precision, by appropriately configuring $w^{b}_{k}$. Note that these spike trains also need to be routed across the chip (Fig. 2), so this does not circumvent synaptic delays.

For the hidden layer, we have chosen those 50 neurons on the chip which responded best to the bias stimulus described above. Only half of the chip was used for these experiments in order to simplify on-chip routing.

The label layer was implemented in software. Spikes produced by the hidden layer were fed into six label neurons simulated with NEST [30]. With this, we essentially broke the hidden$\rightarrow$label$\rightarrow$hidden feedback loop, but as we argued in Sec. IV, it should not significantly affect the classification performance of the network. Furthermore, this allowed a more detailed investigation of the quality of single neurons on the chip, as discussed below. The label assigned by the network to the input image was determined by the label neuron which produced the most spikes during the clamping period.

Fig. 4 shows the classification performance of this setup, along with the spike trains from several exemplary classification runs. The performance of the hardware implementation was

[TABLE]

Within the error margins, this corresponds very well to the reference software simulations (12). The slightly better average classification can be attributed to the explicit selection of the 50 hidden neurons. Indeed, this result not only confirms the robustness of our network model, but also highlights its robustness to various other hardware-induced distortions that we did not explicitly account for, such as parameter noise and crosstalk [5]. Furthermore, this implementation is surprisingly robust even towards few neurons having strongly deviant firing characteristics, as discussed below.

In our network model, hidden neurons are not laterally interconnected. Furthermore, as the label layer was simulated in software, there was also no label-mediated lateral interaction between hidden neurons. Therefore, it was possible to emulate the entire network with one Spikey neuron at a time. In an alternative emulation setup, a sequence of $k=(1,\dots,50)$ emulation runs containing a single hardware neuron was performed. In the $k$th run, the hardware neuron was configured to represent the $k$th hidden neuron by receiving the corresponding input spike train. The output spike trains from these runs were aggregated and fed into the label layer, as before. This experiment was repeated for a subset of 38 out of the 50 selected hardware neurons, with the results plotted as thin bars in Fig. 4A.

The overall performance of each neuron quantifies its quality for the task at hand. The main reason for the differences between the neurons is the shape of their activation function, some of which can be seen in Fig. 2C. Some neurons perform poorly because their activation function is too shallow, thus strongly skewing the sampled distribution. At the other extreme, a very steep activation function is also detrimental, because the resolution of the synaptic weights does not permit an arbitrarily fine-grained tuning of effective weights and biases. Note, in particular, how two of the neurons perform at chance level (Fig. 4A, red bars). However, the existence of such neurons does not appear to have a strong effect on the classification performance of the network as a whole (Fig. 4A, purple vs. green bar).

VI Discussion

One of the most important challenges for analog neuromorphic computing is the design of neural network architectures that are robust to hardware-induced distortions of network dynamics and parameters. In this paper, we have argued that hierarchical spiking sampling networks emulating restricted Boltzmann machines are inherently resistant to such distortions. We have studied three specific distortion mechanisms that are, in general, strongly disruptive to the ongoing computation in sampling LIF networks: synaptic transmission delays, variability of refractory times and saturating membrane conductances. Despite their apparent sensitivity, we have shown how a hierarchical topology shapes the information flow in a way that makes them largely resilient to these effects. The results obtained in software simulations were also confirmed in experiments on an accelerated analog neuromorphic device. Furthermore, in addition to being robust to the studied distortion mechanisms, our network model also displayed an encouraging degree of resilience to other hardware-induced effects which can not be quantified as systematically as the studied ones.

The choice of our neuron model (LIF) was made, on one hand, for analytical tractability, but, more importantly, due to the fact that this model represents a common denominator for many other spiking neuron models (Hodgkin-Huxley, Izhikevich, AdEx). With appropriate parameter choices, all of these models can achieve dynamics that are close to those required for sampling. Furthermore, the LIF model represents a de-facto standard in neuromorphic engineering [4, 5, 6, 7, 8, 9].

Altogether, the observed properties of these networks encourage further theoretical and experimental investigation. Here, we only studied relative performance losses, so we only used a relatively small network and a very small dataset. Software simulations show that larger-scale versions of these networks enable efficient and powerful inference in more complex data spaces [15]. It will be interesting to see whether such large networks remain as robust to hardware-induced distortions as their smaller instantiations studied here. Large-scale accelerated analog devices are already in place [6] and will be able to accomodate these experiments. With our proposed architecture and rather conservative clamping schedule of 1 biological second per image (Fig. 4B), the currently achieved acceleration factor of about ${10}^{4}$ will, for example, enable the classification of the full MNIST dataset within $1\text{\,}\frac{\mathrm{s}}{}$ of wall-clock time.

Although we only used our hierarchical LIF networks for classification, they can also be used as generative models to perform, for example, pattern completion. The nature of the energy landscape in these networks suggests that their generative properties could also be robust to hardware-induced distortions. Since a clear image corresponds to a deep mode in the energy landscape, small distortions in the sampled distribution are unlikely to strongly disrupt the generative properties of the network. Probabilistic switching between different modes when ambiguous input is present can then be facilitated by short-term plasticity as shown in [15], a mechanism that is readily available on several accelerated neuromorphic platforms [5, 6].

In this paper, we have deliberately refrained from further training of the hardware-emulated networks. However, it is expected that training the hardware “in the loop” will significantly improve classification. The idea behind in-the-loop training is to iteratively alternate between a forward pass on hardware, during which the emulated network activity is recorded, and a backward pass in software, where the network parameters are updated by, e.g., error backpropagation. Recent studies have demonstrated, both in software [31] and on analog neuromorphic hardware [14], that the parameter updates need not be precise, but only approximately follow the gradient of the likelihood function. Furthermore, accelerated systems currently in development [32, 33] also implement powerful on-chip learning solutions. Such architectures will not only enable accelerated classification, but, even more importantly, accelerated learning of the network parameters.

Acknowledgments

The first two authors contributed equally to this work. MAP and AS designed and performed the experiments in software and on the Spikey chip. OB designed the software environment for performing the simulations. All authors contributed to writing this paper. We would like to thank Thomas Pfeil and Johannes Bill for technical support with the Spikey chip, as well as Luziwei Leng for providing the model parameters. We further thank Andreas Baumbach, Johannes Bill, Dominik Dold, Carola Fischer, Vitali Karasenko, Akos Kungl and Walter Senn for their invaluable contribution to the final manuscript. This research was supported by EU grant #604102 (Human Brain Project).