An ultra-low-power sigma-delta neuron circuit

TL;DR

This paper introduces a novel sigma-delta neuron circuit in CMOS technology that significantly reduces power consumption while maintaining high signal fidelity, enabling efficient neuromorphic computing and neural network implementation.

Contribution

It proposes a new sigma-delta neuron circuit design that overcomes limitations of existing circuits, with simulation results demonstrating improved signal-to-distortion ratio and ultra-low power consumption.

Findings

Achieves up to 42 dB signal-to-distortion ratio

Consumes orders of magnitude less energy than previous designs

Enables mapping of real-valued RNNs to spiking neural frameworks

Abstract

Neural processing systems typically represent data using leaky integrate and fire (LIF) neuron models that generate spikes or pulse trains at a rate proportional to their input amplitudes. This mechanism requires high firing rates when encoding time-varying signals, leading to increased power consumption. Neuromorphic systems that use adaptive LIF neuron models overcome this problem by encoding signals in the relative timing of their output spikes rather than their rate. In this paper, we analyze recent adaptive LIF neuron circuit implementations and highlight the analogies and differences between them and a first-order sigma-delta feedback loop. We propose a new sigma-delta neuron circuit that addresses some of the limitations in existing implementations and present simulation results that quantify the improvements. We show that the new circuit, implemented in a 1.8 V, 180 nm CMOS process, offers up to 42 dB signal-to-distortion ratio and consumes orders of magnitude lower energy. Finally, we also demonstrate how the sigma-delta interpretation enables mapping of real-valued recurrent neural network to the spiking framework to emphasize the envisioned application of the proposed circuit.

Tables

Table 1. TABLE I: Neuron performance comparison

Project	Tech.	Supply	pJ/spike	Area/neuron
BrainScaleS[20]	65 nm	2.5/1.2 V	200	3372 $\mu m^2$
Neurogrid [3]	180 nm	3 V	8000(est)	1800 $\mu m^2$
ODIN[23]	28 nm	0.55-1.0V	54	74 $\mu m^2$ (est)
TrueNorth[21]	28nm	0.7-1.05V	26(est)	$14.3\mu m^2$
Intel Loihi[22]	14 nm	0.5-1.25V	24 (est)	$14\mu m^2$ (est)
This work	180nm	1.8V	10	2025 $\mu m^2$

Equations

$${\tau }_{mem}\frac{d{I}_{mem}}{dt}=$$

$$+{\alpha }_L\cdot {\Delta }_T\cdot exp(\frac{I_{mem}-\delta }{{\Delta }_T})-s+i$$

$${\tau }_w\frac{ds}{dt}=$$

$$s[n+1]=(1-\alpha )s[n]+\alpha \sigma \left(x[n]W_{in}+s[n]W+b\right)$$

Full text

I Introduction

The effectiveness of artificial neural networks in pattern recognition and classification tasks makes it compelling to use them in ultra-low power applications such as biomedical implants or energy-harvesting smart sensors. However, the computation of neural network models on traditional von Neumann style processors tends to consume considerable amounts of energy, making them nonviable in such power-starved conditions. Event-based neuromorphic processing is an alternative computational approach that tries to address this problem by a combination of in-memory and asynchronous communication techniques [1, 2, 3, 4].

Event-based neuromorphic systems process signals by encoding them as a sequence of asynchronous spikes. The most commonly-used encoding mechanism is rate-coding, where the firing-rate of a LIF neuron is proportional to the amplitude of the input signal. In these conditions, decoding a signal is achieved by simply low-pass filtering the corresponding spike train. To achieve good transmission quality with temporally-changing inputs, it is necessary to set the LIF neuron’s firing rate sufficiently high to encode the input within a short time window. However, transporting a large number of spikes consumes large amounts of power.

Biology has found a solution to this problem by endowing neurons with spike frequency adaptation mechanisms [5, 6]. A neuron model that implements such a mechanism is the Adaptive-Exponential Integrate and Fire (AdExpIF) one. This model has been implemented in silicon in several neuromorphic processors  [7, 8, 9, 2, 10, 11]. In this paper, we observe that the AdExpIF neuron model can be interpreted as a first-order $\Sigma\Delta$ loop. We analyze the AdExpIF silicon neuron circuits in this light and identify issues that affect their power consumption and encoding quality. We then describe an improved $\Sigma\Delta$ neuron circuit that addresses these issues, dramatically lowering power consumption and improving the signal encoding quality. Finally, we present a recurrent neural network simulation to highlight that the $\Sigma\Delta$ interpretation allows us to map floating-point implementation of a recurrent neural network to a spiking neuromorphic one.

II Neuromorphic signal chain

Fig. 1 illustrates the signal flow in a typical recurrently connected neuromorphic system. In this scheme, spike trains generated by the neurons are weighted by the synapse circuits. The weighted sum of currents, $i(t)$ is then integrated by the dendrite circuit F(s). The net synaptic current is further filtered by a neuron leak integrator circuit E(s) before reaching the neuron spike generation block that generates a voltage pulse when its input exceeds a predefined threshold value. The spike generation mechanism introduces non-linearities, $n(t)$ in the signal encoded as a spike train that can be reduced by increasing the firing rate of the neurons.

III The AdExpIF model as a $\Sigma\Delta$ encoder

AdExpIF neuron circuits, such as the one presented in [2] typically use a feedback mechanism to model the spike frequency adaptation mechanism observed in real neurons, as described by the following equations:

[TABLE]

where, $I_{mem}$ and $I_{L}$ are the “membrane potential” and “leak reversal potential” variables that are represented as currents. The term $s$ represents the adaptation current, $i$ the input current, $\tau_{mem}$ the membrane time constant, $\alpha_{L}$ a gain factor, $\delta$ the threshold, $\Delta_{T}$ the slope factor, $\alpha_{s}$ the adaptation coupling parameter and $\tau_{w}$ is the adaptation time constant. The variable $I_{mem}$ is reset to its resting value when $I_{mem}>\delta$.

As $\alpha_{s}$ in the AdExpIF model is set to a very small value, the changes in $s(t)$ are dominated by the spike events: the change in $s(t)$ is that of a first-order low-pass filter responding to a train of impulses. This filter is marked $H(s)$ in Fig. 2. The difference between the input current $i(t)$ and the feedback signal $s(t)$ is filtered by a first-order filter $E(s)$ with gain, $\alpha_{L}$, and time constant, $\tau_{mem}$. When the output of this filter, $I_{mem}$, exceeds the neuron spiking threshold, a spike is generated. This is modeled by the exponential term in Equation 1. With every spike $s(t)$ increases and the difference between the $i(t)$ and $s(t)$ decreases, thus completing the feedback loop. This mechanism is equivalent to an atypical continuous-time $\Delta$ modulation loop [12] with the difference being that the output spikes are unipolar. As described in Section II, input signals arriving to the neuron are encoded as spikes trains and pre-filtered by dendritic or synaptic stages. This filtering operation, $F(s)$ can also be interpreted as a $\Sigma$ stage and the combination of the $\Sigma$ stage and $\Delta$ modulator can be interpreted as a first-order $\Sigma\Delta$ loop. A similar interpretation of the spike response model[13] in neuroscience has also been proposed [14].

Note that the neuron model only generates a spike when the filtered difference or error reaches a spiking threshold. The generated spike trains can be low-pass filtered (for instance, by $F(s)$) to reconstruct the output in an asynchronous manner. There is no quantization in any stage of the signal chain and therefore, no quantization noise is introduced into the signal [15]. However, aliasing and non-linear effects [15, 16] do affect the encoding quality. Depending on the first-order leak to avoid using bipolar pulses also introduces other issues into the loop that are not seen in typical $\Sigma\Delta$ loop. For example, the limit cycle frequency of the proposed loop is proportional to the amplitude of the input signal. In the absence of an input, there is no spike. The gain and time constant of the feedback filters also affect inputs of different frequencies and amplitudes differently.

IV The AdExpIF and $\Sigma\Delta$ neuron circuits

The circuit shown in Fig. 3 is a neuromorphic implementation of the $\Delta$ modulator block of Fig. 2 that is adapted from commonly used silicon neuron circuits in the literature [2, 1]. It is a current-mode circuit where the filter $E(s)$ integrates the difference between the input and feedback outputs. When this difference exceeds the switching threshold of the inverter, it generates a spike. This spike-event induces an increase in the charge held by the feedback capacitance, $C_{fb}$, which in turn increases the feedback current closing the feedback loop. Note that the filters $E(s)$ and $H(s)$ are implemented using Differential Pair Integrator (DPI) circuits [2].

For accurate reconstruction in the $\Sigma\Delta$ loop, it is essential that the pulse trains used for reconstructing the signals be the same as that used in the feedback filter. However, the spike integrated by the feedback filter in the AdExpIF neuron circuit is a narrow digital pulse, which does typically not last long enough for the DPI feedback filter to produce a sufficiently large output. The sub-threshold DPI feedback filter produces a weak response to this narrow pulse. To recover from this, the neuron is forced to spike much more frequently causing the circuit to slew, decreasing performance and increasing power consumption. In this state, the circuit behaves more like a LIF neuron than a $\Sigma\Delta$ loop.

The second problem that mainly affects the energy consumption of the neuron is that the comparator circuit used for generating spikes is an inverter, with a switching threshold of approximately $0.5V_{DD}$. $I_{mem}$ is a slow-changing signal and as long as $I_{mem}$ is slowly rising, the inverter is constantly sinking current. Moreover, the large switching threshold drives the $E(s)$ DPI circuit out of sub-threshold causing non-linear distortions. To address these issues, we modify the circuit by adding a pulse-extender to the spike-generator block and introducing a new low-power current comparator to determine when the integrated current exceeds the (now tunable) spiking threshold. The new $\Sigma\Delta$ circuit schematic is shown in Fig. 4.

IV-A Starved current comparator circuit

The comparator circuit (Fig. 5a) is a current-mode circuit based on [17], with two additional transistors $C7$ and $C8$ added to save power. The input to the comparator circuit is set by the competition between the currents representing the spiking threshold, $\delta$, and $I_{mem}$. When $I_{mem}$ is smaller than $\delta$, the input node to the comparator is close to $V_{DD}$ and vice versa. $I_{mem}$ is a slow-changing signal and so the voltage at “in” falls slowly as the $I_{mem}$ approaches the $\delta$. During this time $C7$ only permits a small current set by the $starve$ bias and $C8$ is off reducing the power wasted through the $C5-C6$ inverter. When the input drops sufficiently, $C8$ turns on and initiates fast switching. This spike event is fed to the pulse extender. The starved current-comparator circuit is responsible for the dramatic reduction in power consumed by the new neuron circuit. A bonus feature is that it allows the spiking threshold of the neuron to be tuned.

IV-B Pulse extender

The $C_{e}$ node of the pulse extender (Fig. 5b) normally sits at $V_{DD}$. As soon as the comparator circuit generates a spike event, the capacitor node is discharged. Immediately after the capacitor is discharged, the comparator input is reset. The time between the generation of a spike event from the comparator to resetting its input is very short. The role of the pulse extender is to stretch the spike event for use in the feedback filter. The circuit achieves this by slowly charged $C_{e}$ by a current source $E1$. When it rises sufficiently, the output switches back to its resting condition. The same bias current is shared by all the pulse extender blocks in the neuromorphic array ensuring that all the filters in the system operate on similar pulses improving the transmission accuracy.

IV-C Regenerative feedback saves power

As soon as the pulse extender registers a spike event, the capacitor $C_{i}$ is fully discharged. This saves power by preventing wasteful current discharge through the $C5-C6$ inverter stage. The pulse extender output also pulls up the input node of the comparator using a regenerative feedback mechanism through $M6$ in Fig. 4. In the absence of this regenerative feedback loop, the comparator input node would be charged only by a small spiking threshold current increasing the power consumption of comparator transistor pair $C5-C6$ very significantly. Similar regenerative feedback loops are also used in the comparator and pulse extender circuits as highlighted with red loops in the circuit diagrams. Any circuit, neuromorphic or otherwise, that interfaces a comparator-like circuit to a slow-changing input could benefit from this technique.

V Simulations

V-A *Regenerative feedback *

To isolate the improved energy-efficiency of the new neuron circuit due to regenerative feedback, we disable the feedback DPI block and apply DC input currents. In this mode, both circuits act as LIF neurons and we observe a linear relationship between the firing rate and input current (Fig. 6a). Observe that the energy consumed per spike in the new $\Sigma\Delta$ circuit is orders of magnitude lower than the AdExpIF model (Fig. 6b). This is because of the three regenerative feedback loops in the $\Sigma\Delta$ circuit that are missing in the AdExpIF circuit. The absence of these loops manifests as a continuous drain of energy in the AdExpIF circuit (Fig. 6c). This is in contrast to the step-like increase in energy consumed in the $\Sigma\Delta$ neuron (Fig. 6d). In most use-cases, the maximum firing rate of a neuron is limited by the communication bandwidth allocated to the neuromorphic system, typically to values around $10^{2}-10^{3}Hz$, where the improved energy-efficiency of the circuit is most useful.

V-B Slewing

Fig. 7 highlights slewing in the AdExpIF neuron due to use of short-duration feedback pulses and the resultant increase in firing rate. In contrast, the pulse extender ensures that each spike event is sufficiently long in the $\Sigma\Delta$ circuit.

V-C Encoding efficiency

In this section, we compare the performance for two neuron circuits when transmitting a $50nA$ sinusoid riding on a $50nA$ DC bias. The bias is essential because the circuits only encode positive inputs. Both circuits were set up with same filter settings. We only show measurements of SDR and power when sweeping the input frequency for different values of gain in the feedback filter as shown in Fig. 8. While the reduced energy consumption ($2\times$ to $40\times$) of the $\Sigma\Delta$ circuit is clearly visible in Fig. 8b and Fig. 8d, the factor of improvement is smaller than in Fig. 6b. This is because of the power consumption in the feedback DPI circuit. We observe that the new $\Sigma\Delta$ neuron circuit also achieves better SDR in Fig. 8a and Fig. 8c.

V-D Reservoir implementation example

A $\Sigma\Delta$ neuron can be used to implement any high-resolution neural network models (ex. 32-bit floating-point) because the feedback signal, $s(t)$, is analogous to the state of the nodes in the network. This is because the event-driven communication model allows a spiking NN to behave like an analog system or at least as a digital system clocked at a very high frequency. (analogous to the equivalence between digital filters and their analog counterparts.) Therefore, it is possible to map a high-resolution neural network model to an equivalent asynchronous spiking one by imposing some constraints. First, the bandwidth of the floating-point neurons should be suitably band-limited. This is not a limitation when interfacing the spiking neuromorphic systems to real-world sensors as they are naurally band-limited. Secondly, the inputs to the spiking system must be scaled such that spiking neurons do not saturate. This is easy to achieve in practice, by use of input gain correction. The mapping mechanism is especially useful to implement RNNs where the improved ability of the $\Sigma\Delta$ circuit to transmit temporally-changing signals is most beneficial. We show an example for this using an ESN [18] with two inputs nodes (including bias), fifty recurrently connected nodes, and a single readout unit (Fig. 9a). The bandwidth constraint is imposed by the retention factor, $0<\alpha<1$, in its state update equation:

[TABLE]

where, $s[n]$ is the reservoir state in time step $n$, $x$ the input, $W_{in}$ the input connectivity matrix, $W$ the recurrent connectivity, $\sigma$ a non-linearity and $b$ the bias. A transient simulation comparing the dynamics of the mapped and original networks is shown in Fig. 9b. Source code is available online [19].

VI Comparison to other neuron implementations

We only report area and power comparisons at a spike generation rate of 300 Hz as literature on temporal data encoding performance of spiking neurons is very scarce (see Table I). The mixed-signal neuron designs in BrainScaleS[20] and Neurogrid[3] offer the closest comparison to this work. TrueNorth[21], Loihi[22] and ODIN[23] are digital systems that use advanced processes, time-multiplexing of neurons and low supply voltages. They are also designed for high input and output data rates and are arguably less suited than the proposed neuron design for low-bandwidth sensor data processing applications. Nevertheless, the proposed $\Sigma\Delta$ neuron circuit consumes the lowest power of the surveyed implementations. Use of advanced processes and smaller supply voltages will further improve the performance of the presented neuron circuit. Finally, we would like to highlight that the presented circuit techniques ($\Sigma\Delta$ model and regenerative feedback) are compatible with and beneficial for many of the neuromorphic systems proposed in literature.

VII Discussion and conclusion

We presented a silicon neuron design implementing a $\Sigma\Delta$ encoder and showed simulations for its targetted use cases. The equivalence to $\Sigma\Delta$ encoders can be further improved by endowing the neuron with the ability to integrate negative errors, using a different log-domain filter design with lesser non-linear distortions, using more accurate circuits to compute the difference between the input and feedback signals. A key novelty of this paper is in the use of regenerative feedback to create a neuromorphic $\Sigma\Delta$ circuit that can operate with orders of magnitude lower energy-consumption than existing neuron implementations, and in its application to real-valued RNNs implemented with spiking neuromorphic systems.