Hyperdimensional Computing with Spiking-Phasor Neurons

TL;DR

This paper explores implementing Vector Symbolic Architectures using spiking neurons to leverage neuromorphic hardware for efficient cognitive computing tasks.

Contribution

It introduces a novel approach to run VSA algorithms on spiking neuron substrates suitable for neuromorphic hardware.

Findings

Demonstrates feasibility of VSA on spiking neurons

Achieves efficient spatial reasoning and symbol binding

Reduces energy consumption compared to traditional methods

Abstract

Vector Symbolic Architectures (VSAs) are a powerful framework for representing compositional reasoning. They lend themselves to neural-network implementations, allowing us to create neural networks that can perform cognitive functions, like spatial reasoning, arithmetic, symbol binding, and logic. But the vectors involved can be quite large, hence the alternative label Hyperdimensional (HD) computing. Advances in neuromorphic hardware hold the promise of reducing the running time and energy footprint of neural networks by orders of magnitude. In this paper, we extend some pioneering work to run VSA algorithms on a substrate of spiking neurons that could be run efficiently on neuromorphic hardware.

Figures

Equations

$${\mathrm{e}}^{\mathrm{i}{\varphi }_1}{\mathrm{e}}^{\mathrm{i}{\varphi }_2}={\mathrm{e}}^{\mathrm{i}\left({\varphi }_1+{\varphi }_2\right)}.$$

$${\mathrm{e}}^{\mathrm{i}{\varphi }_1}{\mathrm{e}}^{-\mathrm{i}{\varphi }_2}={\mathrm{e}}^{\mathrm{i}\left({\varphi }_1-{\varphi }_2\right)}$$

$$\frac{t}{T}=\frac{\varphi }{2\pi },\text{where}T=\frac{1}{\lambda }.$$

$$x^{\prime }=1,x(0)=0,$$

$$\begin{array}[]{lll}p\geq 1-q&\text{and}&p\leq\frac{1}{4}\,,\text{ or}\vspace{1mm}\\ q\geq 1-p&\text{and}&q\leq\frac{1}{4}\ .\end{array}$$

$$f=$$

$$\left(T\otimes S\otimes \rho (P)\right)\oplus \left(P\otimes R\otimes \rho (C)\right)\oplus \left(P\otimes S\otimes \rho (T)\right)$$

$$v=\left(\text{Red}\otimes \text{Square}\otimes X^{1.85}\right)\oplus \left(\text{Blue}\otimes \text{Circle}\otimes X^{-0.65}\right).$$

$$q_1$$

$$q_2$$

Full text

1

\helveticabold

2 Keywords:

VSA, hyperdimensional computing, phasor, spiking, neuromorphic, resonate-and-fire

3 Introduction

Vector Symbolic Architectures (VSAs) offer a way to represent symbols (such as objects or properties) using high-dimensional vectors. Also called Hyperdimensional (HD) computing, these vectors can be combined in different ways to produce other vectors in the same vector space. Amazingly, those new vectors can still be decomposed into their constituent vectors. Hence, a VSA forms a type of compositional language over a set of symbols (called a vocabulary), and is proposed as the key to modelling cognitive processes in the brain (Gayler, 2003). For a survey of VSA approaches, see (Schlegel et al., 2022).

VSAs offer a facility to model cognitive processes within a neural-network substrate. Some implementations use artificial neurons (Pale et al., 2022), and some use spiking-neuron models (Stewart et al., 2011a; Frady et al., 2018). However, the dimensionality of these vectors can be quite large.

Neuromorphic hardware offers a computing platform that can run neural networks at a fraction of the energy cost and running time, compared to other hardware (Thakur et al., 2018). The prospect of running VSA algorithms on neuromorphic hardware seems like an ideal pairing.

To take full advantage of neuromorphic hardware, a spiking VSA needs to be developed. In this paper, we extend the work of Frady and Sommer (2019), and implement a full VSA suite in spiking neurons.

3.1 Vector Symbolic Architectures

A Vector Symbolic Architecture (VSA) is a vector space paired with a small collection of operations: similarity $\odot$, binding $\otimes$, unbinding $\oslash$, bundling $\oplus$, permutation $\rho(\cdot)$, and clean-up.

The similarity operation measures how “close” two vectors are. In a VSA, randomly-chosen, high-dimensional vectors tend to yield a similarity close to zero. For example, consider a vector space $\mathbb{V}$, and two random vectors, $u,v\in\mathbb{V}$. We would expect that $u\odot v\approx 0$, while $v\odot v=u\odot u=1$.

Binding combines two vectors to get another vector that is not similar to either of the two. Thus, if $w=u\otimes v$, then $w\odot u\approx 0$ and $w\odot v\approx 0$. Unbinding does the opposite of binding, so that $w\oslash u\approx v$.

Bundling creates a vector that is still somewhat similar to the two constituent vectors. If $w=u\oplus v$, then $w\odot u\gg 0$, though it will likely not be close to 1. This corruption comes from the fact that some information is lost in the bundling operation. How much is lost depends on the vector space, the specific bundling operation, and how many vectors were bundled together.

Permutation jumbles a vector – in a reversible way – so that the resulting vector is dissimilar from the original. Thus, if $w=\rho(v)$, then $w\odot v\approx 0$, but $\rho^{-1}(w)$ yields $v$ back.

Finally, clean-up takes a vector and restores it to the closest match in the VSA’s vocabulary. The clean-up is helpful because the operations of binding, bundling, etc., tend to corrupt the vectors, and noise can accumulate. Using the noisy vectors in further operations could start to affect their proper function and produce unpredictable behaviours. The clean-up operation can undo that corruption and the clean vector can be used with greater confidence.

3.2 Fourier Holographic Reduced Representation

Fourier Holographic Reduced Representation (FHRR) is a VSA that uses complex-valued unitary vectors to encode symbols. Let $v\in\mathbb{C}^{N}$ be a complex vector. The vector $v$ is unitary if $|v_{k}|=1$ for $k=1,\ldots,N$. Using exponential notation, we can write $v_{k}$ as $\mathrm{e}^{\mathrm{i}\phi_{k}}$, where $\phi_{k}$ is the phase of the complex number. FHRRs are related to Holographic Reduced Representations (HRRs) (Plate, 1995). If the vectors in the FHRR exhibit conjugate symmetry (so that $\phi_{k}=-\phi_{N-k}$), then taking the inverse Fourier transform of $v$ yields a real-valued vector, $\bar{v}\in\mathbb{R}^{N}$.

Binding: In an FHRR, the binding operation is done using element-wise multiplication, the Hadamard product. Since the elements of the FHRR vectors are unit-modulus complex numbers, multiplying them is equivalent to adding their phases,

[TABLE]

The inverse operation of unbinding is elementwise division (or multiplication by the conjugate), as in

[TABLE]

Similarity: If $u$ and $v$ are vectors in an FHRR, then their similarity, $u\odot v$, is computed using the complex inner product, $u\cdot{\bar{v}}$, where $\bar{v}$ is the conjugate of of $v$.

Bundling: In most VSAs, including FHRR, the bundling operation is simply vector addition. Consider adding two unit-modulus complex numbers, $\mathrm{e}^{\mathrm{i}\phi_{a}}+\mathrm{e}^{\mathrm{i}\phi_{b}}$. The resulting complex number will have a phase of $\frac{1}{2}(\phi_{a}+\phi_{b})$, but will probably not be unit-modulus. In the FHRR, the modulus is ignored, and only the phase is kept. Discarding the modulus is a source of information loss, and is one of the reasons that bundling has limitations.

Permutation: Permutation is done by literally permuting the vector elements. If ${\bf P}$ is a permutation matrix, then $\rho(v)={\bf P}v$, and $\rho^{-1}(u)={\bf P}^{-1}(u)$.

Clean-up: In an FHRR, clean-up is often done either by an attractor network (in which the vocabulary vectors are fixed points) (Frady et al., 2018; Frady and Sommer, 2019; Frady et al., 2022), or by an associative memory (Stewart et al., 2011b).

Fractional Binding: The FHRR has an additional operation that is not present in all VSAs. Consider binding a vector $v$ with itself, yielding $v\otimes v$. In the FHRR, $\otimes$ is the Hadamard product, so you can write that as $v^{2}$, where the exponent is applied elementwise. But now one can contemplate using non-integer exponents, such as $v^{1.5}$, or $v^{0.7}$. This is called fractional binding, or Fractional Power Encoding (FPE) (Plate, 1995; Frady et al., 2022). If one of the elements of our unitary vector is $\mathrm{e}^{\mathrm{i}\phi}$, with $-\pi<\phi\leq\pi$, then raising that vector to the exponent $\alpha$ yields a vector with the element $\mathrm{e}^{\mathrm{i}\phi\alpha}$. In other words, the fractional power encoding of $\alpha$ is the same as multiplying the phase by $\alpha$.

4 Methods

As pointed out in (Frady and Sommer, 2019), the phase of a phasor can be represented by the timing of a spike within a cycle. For example, the phase of the complex number $\mathrm{e}^{\mathrm{i}\frac{\pi}{2}}$ can be represented as a spike occurring 0.25 seconds into a cycle that has a period of 1 second, since the ratio $0.25:1=\frac{\pi}{2}:2\pi$. We will call these neurons spiking phasors. In a periodic spiking system, the phase angle, $\phi\in[0,2\pi]$, and the spike offset, $t\in[0,T]$, are directly related by the spiking frequency, $\lambda$ Hz, according to

[TABLE]

Thus, $\phi=\frac{2\pi t}{T}$ converts the spike time, $t$, to phase angle $\phi$. Throughout this paper, we will tend to use phase angles when we refer to spike times, but with the understanding that they are isomorphic to each other.

In the following sections, we describe a series of neuron models that perform the spike-timing computations that correspond to the basic FHRR vector operations.

4.1 Phase-Sum Neuron Model

Recall that binding two unitary vectors involves taking a Hadamard product of the form $\mathrm{e}^{\mathrm{i}\phi_{a}}\mathrm{e}^{\mathrm{i}\phi_{b}}$, which is $\mathrm{e}^{\mathrm{i}(\phi_{a}+\phi_{b})}$. Thus, binding in the FHRR is the same as phase summation. In this section, we describe a model for a spiking neuron such that the phase of its spikes occur at the sum of the phases of the incoming spikes.

Firstly, this neuron (and most other neuron models we will describe) has an integrator variable, $x$, that simply keeps track of the global phase. Recall that these spiking phasor networks are synchronized to a global cycle. If we make $x$ follow the differential equation

[TABLE]

then $x(2\pi)=2\pi$, signifying the end of the cycle, at which point $x$ is reset back to 0. The value of $x$ indicates the phase of the global cycle. The phases of all spiking neurons in the network are with respect to this global cycle.

Let us denote the phases of the incoming spikes as $\phi_{a}$ and $\phi_{b}$, each in the range $[0,2\pi]$. Without loss of generality, assume that $0\leq\phi_{a}\leq\phi_{b}<2\pi$. Figure 1 shows two neurons projecting to a phase-sum neuron, as well as the timeline showing that the phase-sum neuron fires a spike at phase $\phi_{a}+\phi_{b}$.

The general idea behind the phase-sum neuron is that it has an integrator that counts up from the beginning of the cycle until the arrival of the first spike. After the first spike, it holds the integrated value until the arrival of the second spike, at which point it counts down (integrates down) until it reaches 0.

In actuality, the phase addition is accomplished using a combination of two integrators111It can be done with just one integrator, except when the phase sum is greater than one period, and the computations for adjacent periods overlap.. Internal variables $p$ and $q$ act as timers. At the beginning of the cycle, $p=0$. As time progresses, $p$ increases according to the differential equation $p^{\prime}=1$ until the first spike arrives. At the first spike, we set $q=p$, and $q$ is held constant as time continues, with $q^{\prime}=0$. When the second spike arrives, $q$ starts to decrease, following $q^{\prime}=-1$. Once $q=0$, a spike is generated. This process is illustrated in Fig. 2(a). The algorithm for the phase-sum model is articulated in Algorithm 1.

If the sum of the phases is greater than the period of the global cycle, then the computations for adjacent cycles overlap; $p$ has to start counting before $q$ has reached 0. This scenario is shown in Fig. 2(b). This overlap is why we use two integrators, instead of one, allowing $p$ to start timing the arrival of the first spike while $q$ is still finishing the calculation from the previous cycle.

4.2 Phase-Subtraction Neuron Model

In the FHRR, unbinding is the same as binding, but by the conjugate of the vector; this is accomplished by a Hadamard product of the form $\mathrm{e}^{\mathrm{i}\phi_{a}}\mathrm{e}^{-\mathrm{i}\phi_{b}}$, which is $\mathrm{e}^{\mathrm{i}(\phi_{a}-\phi_{b})}$. Hence, unbinding is the same as subtracting phases. The phase-subtraction neuron computes the time that elapses from when the spike arrives from $b$ to when the spike arrives from $a$. This is the only neuron model for which the incoming synapses have to be distinguished from each other; this is a natural consequence of the fact that unbinding is not a commutative operation.

Similar to the phase-sum model, the phase-subtraction model has a global-cycle integrator, $x$. It also has two other integrators that work together to generate a spike at phase $\phi_{a}-\phi_{b}$.

When the spike arrives from $b$, the variable $p$ is set to 0, and starts integrating according to $p^{\prime}=1$. When the spike arrives from $a$, a threshold variable is set, $\theta=p$, essentially recording the elapsed time between the two spikes. Note that this process might span across two adjacent periods of the global cycle. In parallel, the integrator $q$ is set to 0 at the start of each global cycle, and then increases according to $q^{\prime}=1$. When $q$ reaches the threshold, $\theta$, the neuron spikes. These processes are depicted in Fig. 3.

4.3 Phase-Multiplication Neuron Model

Fractional binding is accomplished by multiplying the phase. We can encode the value $\alpha$ in the complex number $\mathrm{e}^{\mathrm{i}\phi}$ using $\mathrm{e}^{\mathrm{i}\phi\alpha}$. If the phase of the input spike was $\phi$, then the phase of the resulting spike is $\phi\alpha$. Note that the input phases are interpreted in the range $[-\pi,\pi]$ instead of $[0,2\pi]$. For example, if $0<\alpha<1$, and $\phi=\frac{\pi}{3}$, then the resulting spike will occur earlier in the cycle.

This neuron model has a baseline cycle integrator, $\hat{x}$. Similar to how the phase $\phi$ is centred on 0, the cycle integrator goes from $-\frac{1}{2}$ to $\frac{1}{2}$ (instead of 0 to 1), but reaches 0 at the same time as the global cycle. Its evolution is governed by the differential equation $\hat{x}^{\prime}=1$. Another integrator, $\hat{p}$, also starts the cycle at $-\frac{1}{2}$, and increases, governed by $\hat{p}^{\prime}=1$.

When a spike arrives (from the single input neuron), the threshold $\theta$ is set to $\alpha\hat{x}$. The neuron generates a spike when $\hat{p}>\theta$. This process is illustrated in Fig. 4.

4.4 Phase-Averaging Neuron Model

In the FHRR, bundling is done by adding the vectors. However, adding unitary vectors usually does not result in a unitary vector. To convert the resulting vector to a unitary vector, the modulus information is discarded. However, the phase still contains some information that can be used.

When adding two complex numbers with the same modulus, the phase of the resulting complex number is the average of the input phases. If the input spikes arrive at 0.2 s and 0.5 s, then the spike of the bundling neuron should occur at 0.35 s. Care should be taken to interpret the incoming phases in the range $[-\pi,\pi]$, rather than $[0,2\pi]$, so that it yields phases closer to 0.

The resulting complex number has to be represented using a spiking phasor neuron, so only the phase of the complex sum is encoded in the neuron.

The time of the average phase is determine using two integrators. Let $p$ be an integrator that starts integrating after the arrival of the first spike, and $q$ be an integrator that starts after the second spike. These integrators reach a value of 1 after one period of the baseline cycle (just as its next spike arrives from the same input neuron). Hence, $(1-p)$ can be thought of as a count-down integrator, essentially tracking the time until the arrival of the next input spike. The midpoint between the 2 spikes is the time when $p=1-q$, or $1-p=q$. But since the furthest apart two spikes can be is half a cycle, then half of that must be less than $\frac{1}{4}$ of a cycle. Hence, the phase-averaging spike should not be more than $\frac{1}{4}$ cycle from either of the incoming spikes. Using this logic, the phase-averaging neuron spikes when

[TABLE]

4.5 Permutation

This is simple; we just re-order the vector components (neurons), usually by circularly shifting all the vector components of $v$ by 1 (so that $\rho(v)_{k}=v_{k+1}$). This new (shifted) vector, $\rho(v)$, is almost orthogonal to the original, $v$, with a cosine similarity close to zero ($\rho(v)\odot v\approx 0$). Recall that their phases are chosen randomly. Hence, the product of the vector with its shifted counterpart will yield complex numbers with phases that are uniformly distributed. Adding up all those components will – on average – be close to zero.

The inverse operation is just as easy. Simply shift the elements back to their original positions.

4.6 Clean-Up Memory

The clean-up memory consists of 2 populations, $G$ and $H$, each containing resonate-and-fire (RF) neurons. The goal of the clean-up memory is to have the neurons in $G$ converge to the nearest vector in the vocabulary. Suppose the vectors in our VSA are $N$-dimensional, so that the population $G$ has $N$ RF neurons. That population projects to $H$, which is a population of $M$ RF neurons, where $M$ is the number of patterns we have in our vocabulary. The connection-weight matrix, $W\in\mathbb{C}^{N\times M}$, between $G$ and $H$ contains the vocabulary vectors, one pattern in each column. The input to $H$ can be thought of as $W^{\mathrm{T}}g$, where $g\in\mathbb{C}^{N}$ is the unitary complex vector representing the spiking phases of the neurons in $G$. This input is like taking the dot-product of $g$ with each vocabulary vector. If $g$ is close to the vector with index $k$, then we would expect the input to $h_{k}$ to be larger than the input to the other hidden neurons. In this sense, each neuron in $H$ represents one of the vocabulary vectors. Importantly, this complex matrix-vector product is implemented using scalar weights and spike delays, as described in (Frady and Sommer, 2019).

The neurons in $H$ all inhibit each other; each time a neuron spikes, it sends strong, inhibitory input to all the other neurons in $H$. Because of this mutual inhibition, the population tends towards a winner-takes-all (one-hot) state.

Finally, $H$ projects back to $G$ using connection weights $W$. That is, each time a neuron in $H$ spikes, it sends to $G$ a weighted copy of the pattern it represents (again, using scalar weights and synaptic delays).

5 Experiments

We implemented the spiking neuron models in the Brian2 (Stimberg et al., 2019). We then tested our spiking-phasor VSA framework on a number of tasks, two of which we describe here.

5.1 State Transition Model

Figure 5 illustrates 3 basic states of a stopwatch, Cleared, Ticking, and Paused. The stopwatch also has 2 buttons, one labelled “R”, which stands for Reset/Record, and one labelled “S”, which stands for Start/Stop. Pressing these buttons can cause the state to transition to a different state. For example, when the stopwatch is in the Cleared state and the S button is pressed (Start), the stopwatch moves to the Ticking state. And when in the Ticking state, pressing the R button keeps it in the Ticking state (to Record a time stamp).

If we represent each of the 3 states with vectors, denoted $C$, $T$, and $P$, and the two actions as vectors, denoted $R$ and $S$, then we can encode a state transition as a vector. For example, a vector $f$ that represent the transition from Cleared to Ticking when the S button is pressed can be constructed using $f=C\otimes S\otimes\rho(T)$. Binding with the vector $\rho(T)$ indicates that the resulting state is Ticking. We can then query $f$ using a vector $q$ of the form $q=\neg C\otimes\neg S$, which asks, “What state will the stopwatch be in if it is in the Cleared state and we press the S button?” The resulting vector will be close to $\rho(T)$, a permuted version of $T$. We unpermute it using $\rho^{-1}$ to get the resulting vector $T$.

There are 6 different state transitions, in total. We can bundle all 6 into a single vector using

[TABLE]

where the $\oplus$ symbol represents bundling, simple vector addition in the case of an FHRR.

We created a 100-dimensional VSA and randomly chose vectors to represent the states and actions, $\{C,T,P,R,S\}$. Each random vector was constructed by randomly choosing 100 phase angles uniformly from the interval $[0,2\pi)$. The vector $f$ was created by adding together the complex-valued vectors in (1).

A population, $F$, was used to store $f$. Each of the 100 neurons in $F$ was firing at the baseline frequency of 10 Hz. The phases of the elements of $f$ were used to set the spike timing of the 100 neurons in $F$. Figure 6 shows the full network, consisting of a total of 705 spiking neurons. The user chooses the state and action, and the output is read from the clean-up memory.

The results of the stopwatch state transition network are shown in Fig. 7. In all cases, the network produced the correct state with a confidence in excess of 99%.

5.2 Spatial Memory Model

The second experiment involves encoding objects and their locations. For this experiment, we will use Spatial Semantic Pointers (SSPs), made possible using fractional exponents in the FHRR. As described in section 4.3, we can raise our FHRR vectors to fractional exponents, and thereby encode that exponent in the resulting vector. This method has been used to encode spatial position (Frady et al., 2018; Lu et al., 2019; Dumont and Eliasmith, 2019).

We randomly generated some 200-dimensional unitary vectors to represent the properties Square, Circle, Red, and Blue. These vectors were generated by randomly sampling 200 phases from a uniform distribution over $[0,2\pi)$. In addition, we used the same method to create another random vector, $X$, to represent a spatial axis. Then we computed a vector that contained multiple objects at different positions,

[TABLE]

The vector $v$ represents a red square at location 1.85, and a blue circle at location -0.65.

We queried $v$ by unbinding it from $X^{1.85}$, essentially asking “What is at location 1.85?” The output from the unbinding was fed into the clean-up memory. The state of the neurons in $G$ were compared to the vocabulary vectors, which included: Square, Circle, Red, Blue, $X$, $Y$, Red$\otimes$Square, Blue$\otimes$Circle, Red$\otimes$Circle, Blue$\otimes$Square. The closest vocabulary vector was Red$\otimes$Square, with a similarity of 0.999.

We also queried $v$ by unbinding it from Red$\otimes$Square, and Blue$\otimes$Circle, essentially asking “Where is the red square?”, and “Where is the blue circle?”

[TABLE]

The resulting populations of spiking neurons were converted to complex vectors and their similarity was calculated against a range of SSPs of the form $X^{x}$, for $-3\leq x\leq 3$. Figure 8 plots the similarity over this range of $x$ values. The entire neural network, including all 3 queries, was implemented using 3,406 spiking neurons.

6 Conclusions

This paper describes our method for implementing FHRR using spiking-phasor neurons and simple integrator neurons. It extends the ideas proposed in (Frady and Sommer, 2019) to operators other than binding. The various vector operations have fairly straight-forward implementations, using integrators to compute the required spike times.

The functionality of these spiking FHRR methods was demonstrated on two tasks, a finite state-transition model, and a spatial memory.

A number of open questions stem from this work. Recall that this spiking-phasor implementation of FHRR does not model the modulus of the complex numbers, just the phase. Thus, all modulus information is discarded. This information loss is particularly relevant after a bundling operation, where the theoretical result would not be unit-modulus, but encoding using spiking-phasor neurons necessitates unit-modulus. It would be interesting to investigate the impact of this loss of information more carefully. For example, what impact does the order in which vectors are added to a bundle have? Can the vectors be chosen in a way that minimizes this loss?

Another direction of further development is a clean-up memory for SSPs. As with any vector in a VSA, sequential binding, bundling, and unbinding deposits noise. This is just as true for an SSP. The clean-up memory proposed in this paper works for a discrete set of vocabulary vectors. In contract, an SSP lies on a 1-dimensional manifold in the vector space, parameterized by the exponent. There is no specific vector to converge to. Instead, a clean-up memory for an SSP would enforce constraints on the phases in the SSP, in a way that pushes the vector back onto the 1-D manifold. This type of coupling has been demonstrated (Ji, 2014), though not on spiking-phasor neurons.

Finally, we plan to implement these methods to run them on a neuromorphic platform. Options include porting our code to Lava to run on the Loihi chip (Davies et al., 2021), or on an FPGA platform (Wang et al., 2014).