Neuro-RAM Unit with Applications to Similarity Testing and Compression in Spiking Neural Networks

TL;DR

This paper introduces a stochastic neural network model with a neuro-RAM module that efficiently performs similarity testing and compression, highlighting the role of randomness and establishing near-optimal tradeoffs in neural computation.

Contribution

It presents a novel neuro-RAM design for similarity testing in spiking neural networks, demonstrating near-optimal tradeoffs and contrasting the computational power of randomized versus deterministic neural models.

Findings

Neuro-RAM achieves sublinear auxiliary neurons for similarity testing.

Randomness offers limited advantage for similarity testing with deterministic threshold gates.

The tradeoff between runtime and network size in neuro-RAM is nearly optimal.

Abstract

We study distributed algorithms implemented in a simplified biologically inspired model for stochastic spiking neural networks. We focus on tradeoffs between computation time and network complexity, along with the role of randomness in efficient neural computation. It is widely accepted that neural computation is inherently stochastic. In recent work, we explored how this stochasticity could be leveraged to solve the `winner-take-all' leader election task. Here, we focus on using randomness in neural algorithms for similarity testing and compression. In the most basic setting, given two $n$-length patterns of firing neurons, we wish to distinguish if the patterns are equal or $\epsilon$-far from equal. Randomization allows us to solve this task with a very compact network, using $O \left (\frac{\sqrt{n}\log n}{\epsilon}\right)$ auxiliary neurons, which is sublinear in the input size. At the heart of our solution is the design of a $t$-round neural random access memory, or indexing network, which we call a neuro-RAM. This module can be implemented with $O(n/t)$ auxiliary neurons and is useful in many applications beyond similarity testing. Using a VC dimension-based argument, we show that the tradeoff between runtime and network size in our neuro-RAM is nearly optimal. Our result has several implications -- since our neuro-RAM can be implemented with deterministic threshold gates, it shows that, in contrast to similarity testing, randomness does not provide significant computational advantages for this problem. It also establishes a separation between feedforward networks whose gates spike with sigmoidal probability functions, and well-studied deterministic sigmoidal networks, whose gates output real number sigmoidal values, and which can implement a neuro-RAM much more efficiently.

Figures

Equations

$$pot(u,t)=\sum _{v\in X\cup Z\cup A}{w}_{v,u}\cdot v^{t-1}-b(u)\text{ and }p(u,t)=\frac{1}{1+e^{-pot(u,t)/\lambda }}$$

$$X_0=\{x_0,...,x_{\sqrt{n}-1}\},...,X_{\sqrt{n}-1}=\{x_{(\sqrt{n}-1)\sqrt{n}},...,x_{n-1}\}.$$

$$\sum _{j=0}^{\sqrt{n}-1}{x}_{i\sqrt{n}+j}\cdot 2^{\sqrt{n}-j}=2\cdot \mathrm{dec}({\bar{X}}_i).$$

$$\mathrm{Pr}[\eta \le x]$$

$$\underset{\mathcal{D}}{\mathrm{Pr}}[{\bar{u}}^1=1]=\mathrm{Pr}[W-\eta \ge 0]$$

$$=\frac{1}{1+e^{-\frac{W-b(u)}{\lambda }}}$$

$$=\mathrm{Pr}[u^1=1].$$

$$\underset{\mathcal{D}}{\mathrm{Pr}}[{\bar{u}}^{i+1}=1|{\bar{B}}_i^i=V]=\mathrm{Pr}[u^{i+1}=1|B_i^i=V].$$

$$1-\frac{1}{2n}$$

$$\underset{X\text{ uniform from }\{0,1{\}}^n}{E}\left[\frac{1}{n}\sum _{Y\in \{0,1{\}}^{\log n}}(1-\mathbb{I}(X,Y))\right]\le \frac{1}{2n}.$$

$$VC(\mathcal{H})\ge \frac{\log |\mathcal{H}|}{\log n+\log e}.$$

$$VC(\mathcal{F})\ge \frac{\log |\mathcal{F}|}{\log n+\log e}=\frac{\log |\mathcal{X}|}{\log n+\log e}\ge \frac{cn}{\log n}$$

$$\mathrm{Pr}[(X_1{)}_{i_t}=(X_2{)}_{i_t}\text{ for all }t\in 1,...,T]\le \frac{1}{n^c}.$$

$$\mathrm{pot}(g_i,t+1)$$

$$=1-2\Vert \mathrm{bin}(i){\Vert }_1+2\sum _{j=0}^{\log n/2-1}\left[\mathrm{bin}(i{)}_j\cdot y_j^t+(\mathrm{bin}(i{)}_j-1)\cdot y_j^t\right]$$

$$=1-2\Vert \mathrm{bin}(i){\Vert }_1+2\sum _{j=0}^{\log n/2-1}\left[\mathrm{bin}(i{)}_j^2+(\mathrm{bin}(i{)}_j-1)\cdot \mathrm{bin}(i{)}_j\right]$$

$$=1-2\Vert \mathrm{bin}(i){\Vert }_1+2\Vert \mathrm{bin}(i){\Vert }_1=1.$$

$$\mathrm{pot}(g_k,t+1)$$

$$\le 1-2\Vert \mathrm{bin}(k){\Vert }_1+2\Vert \mathrm{bin}(k){\Vert }_1-2\le -1$$

$$\mathrm{pot}(e_k,t+2)$$

$$\le -2^{\sqrt{n}}+1+0\le -1$$

$$\mathrm{pot}(d_{j,1},t-1)\le -b(d_{j,1})+w(e_{\mathrm{dec}(Y_1)},d_{j,1})+w(c_l,d_{j,1})\le -2\sqrt{n}-3+2+2\sqrt{n}\le -1$$

$${\color[rgb]{0,0,0}w(d_{j,2},e_{i})=2^{\sqrt{n}-j-1}\text{ and }w(d_{j,3},e_{i})=-2^{\sqrt{n}-j}}.$$

$$\mathrm{pot}(e_i,2)$$

$$+\sum _{j=0}^{\sqrt{n}-1}{X}_{i\sqrt{n}+j}\cdot 2^{\sqrt{n}-j}$$

$$=-2^{\sqrt{n}}+1+\sum _{j=0}^{\sqrt{n}-1}{X}_{i\sqrt{n}+j}\cdot 2^{\sqrt{n}-j}$$

$$\mathrm{pot}(e_i,t)$$

$$+\sum _{j=0}^{\sqrt{n}-1}{X}_{i\sqrt{n}+j}\cdot 2^{\sqrt{n}-j}$$

$$=1-2^{\sqrt{n}}+\sum _{j^{\prime }=1}^{j-1}\left[2^{\sqrt{n}-j^{\prime }-1}-X_{i\sqrt{n}+j^{\prime }}{2}^{\sqrt{n}-j}\right]+\sum _{j=0}^{\sqrt{n}-1}{X}_{i\sqrt{n}+j}\cdot 2^{\sqrt{n}-j}$$

$$=1-2^{\sqrt{n}}+\sum _{j^{\prime }=0}^{j-1}{2}^{\sqrt{n}-j^{\prime }-1}+\sum _{j^{\prime }=j}^{\sqrt{n}-1}{X}_{i\sqrt{n}+j^{\prime }}{2}^{\sqrt{n}-j}$$

$$=1-2^{\sqrt{n}-j}+\sum _{j^{\prime }=j}^{\sqrt{n}-1}{X}_{i\sqrt{n}+j^{\prime }}{2}^{\sqrt{n}-j}$$

Full text

It is widely accepted that neural spike responses, and neural computation in general, is inherently stochastic. In recent work, we explored how this stochasticity could be leveraged to solve the ‘winner-take-all’ leader election task. Here, we focus on using randomness in neural algorithms for similarity testing and compression. In the most basic setting, given two $n$-length patterns of firing neurons, we wish to distinguish if the patterns are equal or $\epsilon$-far from equal.

Randomization allows us to solve this task with a very compact network, using $O\left(\frac{\sqrt{n}\log n}{\epsilon}\right)$ auxiliary neurons, which is sublinear in the input size. At the heart of our solution is the design of a $t$-round neural random access memory, or indexing network, which we call a neuro-RAM. This module can be implemented with $O(n/t)$ auxiliary neurons and is useful in many applications beyond similarity testing – e.g. we discuss its application to compression via random projection.

Using a VC dimension-based argument, we show that the tradeoff between runtime and network size in our neuro-RAM is nearly optimal. To the best of our knowledge, we are the first to apply these techniques to stochastic spiking networks. Our result has several implications – since our neuro-RAM can be implemented with deterministic threshold gates, it shows that, in contrast to similarity testing, randomness does not provide significant computational advantages for this problem. It also establishes a separation between feedforward networks whose gates spike with sigmoidal probability functions, and well-studied deterministic sigmoidal networks, whose gates output real number sigmoidal values, and which can implement a neuro-RAM much more efficiently.

1 Introduction

Biological neural networks are arguably the most fascinating distributed computing systems in our world. However, while studied extensively in the fields of computational neuroscience and artificial intelligence, they have received little attention from a distributed computing perspective. Our goal is to study biological neural networks through the lens of distributed computing theory. We focus on understanding tradeoffs between computation time, network complexity, and the use of randomness in implementing basic algorithmic primitives, which can serve as building blocks for high level pattern recognition, learning, and processing tasks.

Spiking Neural Network (SNN) Model

We work with biologically inspired spiking neural networks (SNNs) [Maa96, Maa97, GK02, Izh04], in which neurons fire in discrete pulses in synchronous rounds, in response to a sufficiently high membrane potential. This potential is induced by spikes from neighboring neurons, which can have either an excitatory or inhibitory effect (increasing or decreasing the potential). As observed in biological networks, neurons are either strictly inhibitory (all outgoing edge weights are negative) or excitatory. As we will see, this restriction can significantly affect the power of these networks.

A key feature of our model is stochasticity – each neuron is a probabilistic threshold unit, spiking with probability given by applying a sigmoid function to its potential. While a rich literature focuses on deterministic circuits [MP69, HT*+*86] we employ a stochastic model as it is widely accepted that neural computation is stochastic [AS94, SN94, FSW08].

Computational Problems in SNNs

We consider an $n$-bit binary input vector $X$, which represents the firing status of a set of input neurons. Given a (possibly multi-valued) function $f:\{0,1\}^{n}\to\{0,1\}^{m}$, we seek to design a network of spiking neurons that converges to an output vector $Z=f(X)$ (or any $Z\in f(X)$ if $f$ is multi-valued) as quickly as possible using few auxiliary (non-input or output) neurons.

The number of auxiliary neurons used corresponds to the “node complexity” of the network [HH94]. Designing circuits with small node complexity has received a lot of attention – e.g., the work of [FSS84] on PARITY and [All89] on $AC_{0}$. Much less is known, however, on what is achievable in spiking neural networks. For most of the problems we study, there is a trivial solution that uses $\Theta(n)$ auxiliary neurons for inputs of size $n$. Hence, we primarily focus on designing sublinear size networks – with $n^{1-c}$ auxiliary neurons for some $c$.

Past Work: WTA

Recently, we studied the ‘winner-take-all’ (WTA) leader election task in SNNs [LMP17]. Given a set of firing input neurons, the network is required to converge to a single firing output – corresponding to the ‘winning’ input. In that work, we critically leveraged the noisy behavior of our spiking neuron model: randomness is key in breaking the symmetry between initially identical firing inputs.

This Paper: Similarity Testing and Compression

In this paper, we study the role of randomness in a different setting: for similarity testing and compression. Consider the basic similarity testing problem: given $X_{1},X_{2}\in\{0,1\}^{n}$, we wish to distinguish the case when $X_{1}=X_{2}$ from the case when the Hamming distance between the vectors is large – i.e., $\operatorname{d_{H}}(X_{1},X_{2})\geq\epsilon n$ for some parameter $\epsilon$. This problem can be solved very efficiently using randomness – it suffices to sample $O(\log n/\epsilon)$ indices and compare $X_{1}$ and $X_{2}$ at these positions to distinguish the two cases with high probability. Beyond similarity testing, similar compression approaches using random input subsampling or hashing can lead to very efficient routines for a number of data processing tasks.

1.1 A Neuro-RAM Unit

To implement the randomized similarity testing approach described above, and to serve as a foundation for other random compression methods in spiking networks, we design a basic indexing module, or random access memory, which we call a neuro-RAM. This module solves:

Definition 1** (Indexing).**

Given $X\in\{0,1\}^{n}$ and $Y\in\{0,1\}^{\log n}$ which is interpreted as an integer in $\{0,...,n-1\}$, the indexing problem is to output the value of the $Y^{th}$ bit of $X$111Here, and throughout, for simplicity we assume $n$ is a power of $2$ so $\log n$ is an interger..

Our neuro-RAM uses a sublinear number of auxiliary neurons and solves indexing with high probability on any input. We focus on characterizing the trade-off between the convergence time and network size of the neuro-RAM, giving nearly matching upper and lower bounds.

Generally, our results show that a compressed representation (e.g., the index $Y$) can be used to access a much larger datastore (e.g., $X$), using a very compact neural network. While binary indexing is not very ‘neural’ we can imagine similar ideas extending to more natural coding schemes used, for example, for memory retrieval, scent recognition, or other tasks.

Relation to Prior Work

Significant work has employed random synaptic connections between neurons – e.g., the Johnson-Lindenstrauss compression results of [AZGMS14] and the work of Valiant [Val00]. While it is reasonable to assume that the initial synapses are random, biological mechanisms for changing connectivity (functional plasticity) act over relatively large time frames and cannot provide a new random sample of the network for each new input. In contrast, stochastic spiking neurons do provide fresh randomness to each computation. In general, transforming of a network with $m$ possible random edges to a network with fixed edges and stochastic neurons requires $\Omega(m)$ auxiliary neurons and thus fails to fulfill our sublinearity goal, as there is typically at least one possible outgoing edge from each input. Our neuro-RAM can be thought of as improving the naive simulation – by reading a random entry of an input, we simulate a random edge from the specified neuron. Beyond similarity testing, we outline how our result can be used to implement Johnson-Lindenstrauss compression similar to [AZGMS14] without assuming random connectivity.

1.2 Our Contributions

1.2.1 Efficient Neuro-RAM Unit

Our primary upper bound result is the following:

Theorem 2** ($t$-round Neuro-RAM).**

For every integer $t\leq\sqrt{n}$, there is a (recurrent) SNN with $O(n/t)$ auxiliary neurons that solves the indexing problem in $t$ rounds with high probability. In particular, there exists a neuro-RAM unit that contains $O(\sqrt{n})$ auxiliary neurons and solves the indexing problem in $O(\sqrt{n})$ rounds.

Above, and throughout the paper ‘with high probability’ or w.h.p. to denotes with probability at least $1-1/n^{c}$ for some constant $c$. Theorem 2 is proven in Section 3.

Neuro-RAM Construction

The main idea is to first ‘encode’ the firing pattern of the input neurons $X$ into the potentials of $t$ neurons. These encoding neurons will spike with some probability dependent on their potential. However, simply recording the firing rates of the neurons to estimate this probability is too inefficient. Instead, we use a ‘successive decoding strategy’, in which the firing rates of the encoding neurons are estimated at finer and finer levels of approximation, and adjusted through recurrent excitation or inhibition as decoding progresses. The strategy converges in $O(n/t)$ rounds – the smaller $t$ is the more information is contained in the potential of a single neuron, and the longer decoding takes.

Theorem 2 shows a significant separation between our networks and traditional feedforward circuits where significantly sublinear sized indexing units are not possible.

Fact 3** (See Lower Bounds in [Koi96]).**

A circuit solving the indexing problem that consists of AND/OR gates connected in a feedforward manner requires $\Theta(n)$ gates. A feedforward circuit using linear threshold gates requires $\Theta(n/\log n)$ gates.

We note, however, that our indexing mechanism does not exploit the randomness of the spiking neurons, and in fact can also be implemented with deterministic linear threshold gates. Thus, the separation between Theorem 2 and Fact 3 is entirely due to the recurrent (non-feedforward) layout of our network. Since any recurrent network using $O(m)$ neurons and converging in $t$ rounds can be ‘unrolled’ into a feedforward circuit using $O(mt)$ neurons, Fact 3 shows that the tradeoff between network size and runtime in Theorem 2 is optimal up to a $\log n$ factor, as long as we use our spiking neurons in a way that can also be implemented with deterministic threshold gates. However, it does not rule out improvements using more sophisticated randomized strategies.

1.2.2 Lower Bound for Neuro-RAM in Spiking Networks

Surprisingly, we are able to show that despite the restricted way in which we use our spiking neuron model, significant improvements are not possible:

Theorem 4** (Lower Bound for Neuro-RAM in SNNs).**

Any SNN that solves indexing in $t$ rounds with high probability in our model must use at least $\Omega\left(\frac{n}{t\log^{2}n}\right)$ auxiliary neurons.

Theorem 4, whose proof is in Section 4, shows that the tradeoff in Theorem 2 is within a $\log^{2}n$ factor of optimal. It matches the lower bound of Fact 3 for deterministic threshold gates up to a $\log n$ factor, showing that there is not a significant difference in the power of stochastic neurons and deterministic gates in solving indexing.

Reduction from SNNs to Deterministic Circuits

We first argue that the output distribution of any SNN is identical to the output distribution of an algorithm that first chooses a deterministic threshold circuit from some distribution and then applies it to the input. This is a powerful observation as it lets us apply Yao’s principle: an SNN lower bound can be shown via a lower bound for deterministic circuits on any input distribution [Yao77].

Deterministic Circuit Lower Bound via VC Dimension

We next show that any deterministic circuit that succeeds with high probability on the uniform input distribution cannot be too small. The bound is via a VC dimension-based argument, which extends the work of [Koi96] on indexing circuits. As far as we are aware, we are the first to give a VC dimension-based lower bound for probabilistic and biologically plausible network architectures and we hope our work significantly expands the toolkit for proving lower bounds in this area. In contrast to our lower bounds on the WTA problem [LMP17], which rely on indistinguishability arguments based on network structure, our new techniques allow us to give more general bounds for any network architecture.

Separation of Network Models

Aside from showing that randomness does not give significant advantages in constructing a neuro-RAM (contrasting with its importance in WTA and similarity testing), our proof of Theorem 4 establishes a separation between feedforward spiking networks and deterministic sigmoidal circuits. Our neurons spike with probability computed as a sigmoid of their membrane potential. In sigmoidal circuits, neurons output real numbers, equivalent to our spiking probabilities. A neuro-RAM can be implemented very efficiently in these networks:

Fact 5** (See [Koi96], along with [Maa97] for similar bounds).**

There is a feedforward sigmoidal circuit solving the indexing problem using $O(n^{1/2})$ gates.222Note that [MSS91] shows that general deterministic sigmoidal circuits can be simulated by our spiking model. However, the simulation blows up the size of the circuit size by $\sqrt{n}$, giving $\Theta(n)$ auxiliary neurons.

In contrast, via an unrolling argument, the proof of Theorem 4 shows that any feedforward spiking network requires $\Omega\left(\frac{n}{\log^{2}n}\right)$ gates to solve indexing with high probability.

It has been shown that feedforward sigmoidal circuits can significantly outperform standard feedforward linear threshold circuits [MSS91, Koi96]. However, previously it was not known that restricting gates to spike with a sigmoid probability function rather than output the real value of this function significantly affected their power. Our lower bound, along with Fact 5, shows that in some cases it does. This separation highlights the importance of modeling spiking neuron behavior in understanding complexity tradeoffs in neural computation.

1.2.3 Applications to Randomized Similarity Testing and Compression

As discussed, our neuro-RAM is widely applicable to algorithms that require random sampling of inputs. In Section 5 we discuss our main application, to similarity testing – i.e., testing if $X_{1}=X_{2}$ or if $\operatorname{d_{H}}(X_{1},X_{2})\geq\epsilon n$. It is easy to implement an exact equality tester using $\Theta(n)$ auxiliary neurons. Alternatively, one can solve exact equality with three auxiliary neurons using mixed positive and negative edge weights for the outgoing edges of inputs. However this is not biologically plausible – neurons typically have either all positive (excitatory) or all negative (inhibitory) outgoing edges, a restriction included in our model. Designing sublinear sized exact equality testers under this restriction seems difficult – simulating the three neuron solution requires at least $\Theta(n)$ auxiliary neurons – $\Theta(1)$ for each input.

By relaxing to similarity testing and applying our neuro-RAM, we can achieve sublinear sized networks. We can use $\Theta(\log n/\epsilon)$ neuro-RAMs, each with $O(\sqrt{n})$ auxiliary neurons to check equality at $\Theta(\log n/\epsilon)$ random positions of $X_{1}$ and $X_{2}$ distinguishing if $X_{1}=X_{2}$ or if $\operatorname{d_{H}}(X_{1},X_{2})\geq\epsilon n$ with high probability. This is the first sublinear solution for this problem in the spiking neural networks. In Section 5, we discuss possible additional applications of our neuro-RAM to Johnson-Lindenstrauss random compression, which amounts to multiplying the input by a sparse random matrix – a generalization of input sampling.

2 Computational Model and Preliminaries

2.1 Network Structure

We now give a formal definition of our computational model. A Spiking Neural Network (SNN) $\operatorname{\mathcal{N}}=\langle X,Z,A,w,b\rangle$ consists of $n$ input neurons $X=\{x_{1},\ldots,x_{n}\}$, $m$ output neurons $Z=\{z_{1},\ldots,z_{m}\}$, and $\ell$ auxiliary neurons $A=\{a_{1},...,a_{\ell}\}$. The directed, weighted synaptic connections between $X$, $Z$, and $A$ are described by the weight function $w:[X\cup Z\cup A]\times[X\cup Z\cup A]\rightarrow\mathbb{R}$. A weight $w(u,v)=0$ indicates that a connection is not present between neurons $u$ and $v$. Finally, for any neuron $v$, $\operatorname{b}(v)\in\mathbb{R}_{\geq 0}$ is the activation bias – as we will see, roughly, $v$’s membrane potential must reach $\operatorname{b}(v)$ for a spike to occur with good probability.

The weight function defining the synapses in our networks is restricted in a few notable ways. The in-degree of every input neuron $x_{i}$ is zero. That is, $w(u,x)=0$ for all $u\in[X\cup Z\cup A]$ and $x\in X$. This restriction bears in mind that the input layer might in fact be the output layer of another network and so incoming connections are avoided to allow for the composition of networks in higher level modular designs. Additionally, each neuron is either inhibitory or excitatory: if $v$ is inhibitory, then $w(v,u)\leq 0$ for every $u$, and if $v$ is excitatory, then $w(v,u)\geq 0$ for every $u$. All input and output neurons are excitatory.

2.2 Network Dynamics

An SNN evolves in discrete, synchronous rounds as a Markov chain. The firing probability of every neuron at time $t$ depends on the firing status of its neighbors at time $t-1$, via a standard sigmoid function, with details given below.

For each neuron $u$, and each time $t\geq 0$, let $u^{t}=1$ if $u$ fires (i.e., generates a spike) at time $t$. Let $u^{0}$ denote the initial firing state of the neuron. Our results will specify the initial input firing states $x_{j}^{0}=1$ and assume that $u^{0}=0$ for all $u\in[Z\cup A]$. For each non-input neuron $u$ and every $t\geq 1$, let $pot(u,t)$ denote the membrane potential at round $t$ and $p(u,t)$ denote the corresponding firing probability ($\operatorname*{\mathrm{Pr}}[u^{t}=1]$). These values are calculated as:

[TABLE]

where $\lambda>0$ is a temperature parameter, which determines the steepness of the sigmoid. It is easy to see that $\lambda$ does not affect the computational power of the network. A network can be made to work with any $\lambda$ simply by scaling the synapse weights and biases appropriately.

For simplicity we assume throughout that $\lambda=\frac{1}{\Theta(\log n)}$. Thus by (1), if $pot(u,t)\geq 1$, then $u^{t}=1$ w.h.p. and if $pot(u,t)\leq-1$, $u^{t}=0$ w.h.p. (recall that w.h.p. denotes with probability at least $1-1/n^{c}$ for some constant $c$). Aside from this fact, the only other consequence of (1) we use in our network constructions is that $pot(u,t)=0\implies p(u,t)=1/2$. That is, we will use our spiking neurons entirely as random threshold gates, which fire w.h.p. when the incoming potential from their neighbors’ spikes exceeds $\operatorname{b}(u)$, don’t fire w.h.p. when the potential is below $\operatorname{b}(u)$, and fire randomly when the input potential equals the bias. It is an interesting open question if there are any problems which require using the full power of the sigmoidal probability function.

2.3 Additional Notation

For any vector $x$ we let $x_{i}$ denote the value at its $i^{th}$ position, starting from $x_{0}$. Given binary $x\in\{0,1\}^{n}$, we use $\operatorname{dec}(x)$ to indicate the integer encoded by $x$. That is, $\operatorname{dec}(x)=\sum_{i=0}^{n-1}x_{i}\cdot 2^{i}$. Given an integer $x$ we use $\operatorname{bin}(x)$ to denote its binary encoding, where the number of digits used in the encoding will be clear from context. We will often think of the firing pattern of a set of neurons as a binary string. If $B=\{y_{1},...,y_{m}\}$ is a set of $m$ neurons then $B^{t}\in\{0,1\}^{m}$ is the binary string corresponding to their firing pattern at time $t$. Since the input is typically fixed for some number of rounds, we often just write $X$ to refer to the $n$-bit string corresponding to the input firing pattern.

Boolean Circuits.

We mention that SNNs are similar to boolean circuits, which have received enormous attention in theoretical computer science. A circuit consists of gates (e.g., threshold gates, probabilistic threshold gates) connected in a directed acyclic graph. This restriction means that a circuit does not have feedback connections or self-loops, which we do use in our SNNs. While we do not work with circuits directly, for our lower bound, we show a transformation from an SNN to a linear threshold circuit. We sometimes refer to circuits as feedforward networks, indicating that their connections are cycle-free.

3 Neuro-RAM Network

In this section we prove our main upper bound:

Theorem 6** (Efficient Neuro-RAM Network).**

There exists an SSN with $O(\sqrt{n})$ auxiliary neurons that solves indexing in $5\sqrt{n}$ rounds. Specifically, given inputs $X\in\{0,1\}^{n}$, and $Y\in\{0,1\}^{\log n}$, which are fixed for all rounds $t\in\{0,...,5\sqrt{n}\}$, the output neuron $z$ satisfies: if $X_{\operatorname{dec}(Y)}=1$ then $z^{5\sqrt{n}}=1$ w.h.p. Otherwise, if $X_{\operatorname{dec}(Y)}=0$, $z^{5\sqrt{n}}=0$ w.h.p.

Theorem 6 easily generalizes to other network sizes, giving Theorem 2, which states the general size-time tradeoff. Here we discuss the basic construction and intuition behind our network construction. The full details and proof are given in Appendice A.1 and A.2.

We divide the $n$ input neurons X into $\sqrt{n}$ buckets each containing $\sqrt{n}$ neurons333Throughout we assume for simplicity that $n=2^{2m}$ for some integer $m$. This ensures that $\sqrt{n}$, $\log n$, and $\log\sqrt{n}$ are integers. It will be clear that if this is not the case, we can simply pad the input, which only affects our time and network size bounds by constant factors.:

[TABLE]

Throughout, all our indices start from [math]. We encode the firing pattern of each bucket $X_{i}$ via the potential of a single neuron $e_{i}$. Set {\color[rgb]{0,0,0}w(x_{i\sqrt{n}+j},e_{i})=2^{\sqrt{n}-j}} for all $i,j\geq 0$. In this way, for every round $t$, the total potential contributed to $e_{i}$ by the firing of the inputs in bucket $X_{i}$ is equal to:

[TABLE]

where $\bar{X}_{i}$ is the reversal of $X_{i}$ and $\operatorname{dec}(\cdot)$ gives the decimal value of a binary string, as defined in the preliminaries. We set $\operatorname{b}(e_{i})=2^{\sqrt{n}+2}+2^{\sqrt{n}}-1$. We will see later why this is an appropriate value. We defer detailed discussion of the remaining connections to $e_{i}$ for now, first giving a general description of the network construction.

In addition to the encoding neurons $e_{0},...,e_{\sqrt{n}-1}$, we have decoding neurons $d_{0,k},...,d_{\sqrt{n}-1,k}$ for $k=1,2,3$ ($3\sqrt{n}$ neurons total). The idea is to select a bucket $X_{i}$ (via $e_{i}$) using the first $\log\sqrt{n}=\frac{\log n}{2}$ bits in the index $Y$. Let $Y_{1}\mathbin{\stackrel{{\scriptstyle\rm def}}{{=}}}\{y_{0},...,y_{\frac{\log n}{2}-1}\}$ and $Y_{2}\mathbin{\stackrel{{\scriptstyle\rm def}}{{=}}}\{y_{\frac{\log n}{2}},...,y_{\log n-1}\}$ be the higher and lower order bits of $Y$ respectively. It is not hard to see that using $O(\sqrt{n})$ neurons we can construct a network that processes $Y_{1}$ and uses it to select $e_{i}$ with $i=\operatorname{dec}(Y_{1})$. When a bucket is selected, the potential of any $e_{j}$ with $j\neq\operatorname{dec}(Y_{1})$ is significantly depressed compared to that of $e_{i}$ and so after this selection stage, only $e_{i}$ fires.

We will then use the decoding neurons to ‘read’ each bit of the potential encoded in $e_{i}$. The final output is selected from each of these bits using the lower order bits $Y_{2}$, which can again be done efficiently with $O(\sqrt{n})$ neurons. We call this phase the decoding phase since the bucket neuron $e_{i}$ encodes the value (in decimal) of its bucket $X_{i}$, and we need to decode from that value the bit of the appropriate neuron inside that bucket.

The decoding process works as follows: initially, $e_{i}$ will fire only if the first bit of bucket $i$ is on. Note that the weight from this bit to $e_{i}$ is $2^{\sqrt{n}}$ and thus more than double the weight from any other input bit. Thus, by appropriately setting $\operatorname{b}(e_{i})$, we can ensure that the setting of this single bit determines if $e_{i}$ fires initially.

If the first bit is the correct bit to output (i.e. if the last $\frac{\log n}{2}$ bits of the index $Y_{2}$ encode position [math]), this will trigger the output $z$ to fire. Otherwise, we iterate. If $e_{i}$ in fact fired, this triggers inhibition that cancels out the potential due to the first bit of bucket $i$. Thus, $e_{i}$ will now only fire if the second bit of $X_{i}$ is on. If $e_{i}$ did not fire, the opposite will happen. Further excitation will be given to $e_{i}$ again ensuring that it can fire as long as the second bit of $X_{i}$ is on. The network iterates in this way, successively reading each bit, until we reach the one encoded by $Y_{2}$ and the output fires. The first decoding neuron for position $j$, $d_{j,1}$, is responsible to triggering the output to fire if $j$ is the correct bit encoded by $Y_{2}$. The second decoding neuron $d_{j,2}$ is responsible for providing excitation when $e_{i}$ does not fire. Finally, the third decoding neuron $d_{j,3}$ provides inhibition when $e_{i}$ does fire.

In Appendix A.1, we describe the first stage in which we use the first $\log n/2$ index bits to select the bucket to which the desired index belongs to.

In Appendix A.2, we discuss the second phase where we use the last $\log n/2$ bits of $Y$, to select the desired index inside the bucket $i$. Our success decoding process is synchronized by a clock mechanism, shown in Appendix A.2.1. This clock mechanism consists of chain of $\Theta(\sqrt{n})$ neurons that govern the timing of the $\Theta(\sqrt{n})$ steps of our decoding scheme. Roughly speaking, traversing the $\sqrt{n}$ bits of the chosen $i^{th}$ bucket from left to right, we spend $O(1)$ rounds checking if the current index is the one encoded by $Y_{2}$. If yes, we output the value at that index and if not, the clock will “tick” and we move to the next candidate. This successive decoding scheme is explained in Appendix A.2.2.

Note that our model and the proof of Theorem 6 assume that no auxiliary neurons or the output neuron fire in round [math]. However, in applications it will often be desirable to run the Neuro-RAM for multiple inputs, with execution not necessarily starting at round [math]. We can easily add a mechanism that ‘clears’ the network once it outputs, giving:

Observation 7** (Running Neuro-RAM for Multiple Inputs).**

The Neuro-RAM of Theorem 6 can be made to run correctly given a sequence of multiple inputs.

4 Lower Bound for Neuro-RAM in Spiking Networks

In this section, we show that our neuro-RAM construction is nearly optimal. Specifically:

Theorem 8**.**

Any SNN solving indexing with probability $\geq 1-\frac{1}{2n}$ in $t$ rounds must use $\ell=\Omega\left(\frac{n}{t\log^{2}n}\right)$ auxiliary neurons.

This result matches the lower bound for deterministic threshold gates of Fact 3 up to a $\log n$ factor, demonstrating that the use of randomness cannot give significant runtime advantages for the indexing problem. We note that even if one just desires a constant (e.g. $2/3$) probability of success – the lower bound applies. By replicating any network with success probability $2/3$, $\Theta(\log n)$ times and taking the majority output (which can be computed with just a single additional auxiliary neuron), we obtain a network that solves the problem w.h.p. We thus have:

Corollary 9**.**

Any SNN solving indexing with probability $\geq 2/3$ in $t$ rounds must use $\ell=\Omega\left(\frac{n}{t\log^{3}n}\right)$ auxiliary neurons.

The proof of Theorem 8 proceeds in a number of steps, which we overview here.

4.1 High Level Approach and Intuition

Reduction to Deterministic Indexing Circuit.

We first observe that a network with $\ell$ auxiliary neurons solving the indexing problem in $t$ rounds can be unrolled into a feedforward circuit with $t$ layers and $\ell$ neurons per layer. We then show that the output distribution of a feedforward stochastic spiking circuit is identical to the output distribution if we first draw a deterministic linear threshold circuit (still with $t$ layers and $\ell$ neurons per layer) from a certain distribution, and evaluate our input using this random circuit.

This equivalence is powerful since it allows us to apply Yao’s minimax principal [Yao77]: assuming the existence of a feedforward SNN solving indexing with probability $\geq 1-\frac{1}{2n}$, given any distribution of the inputs $X,Y$, there must be some deterministic linear threshold circuit $\mathcal{N}_{D}$ which solves indexing with probability $\geq 1-\frac{1}{2n}$ over this distribution.

If we consider the uniform distribution over $X,Y$, this success probablity ensures via an averaging argument that for at least $1/2$ of the $2^{n}$ possible values of $X$, $\mathcal{N}_{D}$ succeeds for at least a $1-\frac{1}{2n}$ fraction of the possible $Y$ inputs. Note, however, that the $Y$ can only take on $n$ possible values – thus this ensures that for $1/2$ the possible values of $X$, $\mathcal{N}_{D}$ succeeds for all possible values of the index $Y$. Let $\mathcal{X}$ be the set of ‘good inputs’ for which $\mathcal{N}_{D}$ succeeds.

Lower Bound for Deterministic Indexing on a Subset of Inputs.

We have now reduced our problem to giving a lower bound on the size of a deterministic linear threshold circuit which solves indexing on an arbitrary subset $\mathcal{X}$ of $\frac{1}{2}\cdot 2^{n}=2^{n-1}$ inputs. We do this using VC dimension techniques inspired by the indexing lower bound of [Koi96].

The key idea is to observe that if we fix some input $X\in\mathcal{X}$, then given $Y$, $\mathcal{N}_{D}$ evaluates the function $f_{X}:\{0,1\}^{\log n}\rightarrow\{0,1\}$, whose truth table is given by $X$. Thus $\mathcal{N}_{D}$ can be viewed as a circuit for evaluating any function $f_{X}(Y)$ for $X\in\mathcal{X}$, where the $X$ inputs are ‘programmable parameters’, which effectively change the thresholds of some gates.

It can be shown that the VC dimension of the class of functions computable by a fixed a linear threshold circuit with $m$ gates and variable thresholds is $O(m\log m)$. Thus for a circuit with $t$ layers and $\ell$ gates per layer, the VC dimension is $O(\ell t\log(\ell t))$ [BH89]. Further, as a consequence of Sauer’s Lemma [Sau72, She72, AB09], defining the class of functions $\mathcal{F}=\{f_{X}\text{ for any }X\in\mathcal{X}\}$, since $|\mathcal{F}|=|\mathcal{X}|=2^{n-1}$, we have $VC(\mathcal{F})=\Theta(n/\log n)$. These two VC dimension bounds, in combination with the fact that we know $\mathcal{N}_{D}$ can compute any function in $\mathcal{F}$ if its input bits are fixed appropriately, imply that $\ell t\cdot\log(\ell t)=\Omega(n/\log n)$. Rearranging gives $\ell=\Omega\left(\frac{n}{t\log^{2}n}\right)$, completing Theorem 8.

4.2 Reduction to Deterministic Indexing Circuit

We now give the argument explained above in detail, first describing how any SNN that solves indexing w.h.p. implies the existence of a deterministic feedforward linear threshold circuit which solves indexing for a large fraction of possible inputs $X$.

Lemma 10** (Conversion to Feedforward Network).**

Consider any SNN $\mathcal{N}$ with $\ell$ auxiliary neurons, which given input $X\in\{0,1\}^{n}$ that is fixed for rounds $\{0,...,t\}$, has output $z$ satisfying $\operatorname*{\mathrm{Pr}}[z^{t}=1]=p$. Then there is a feedforward SNN $\mathcal{N}_{F}$ (an SNN whose directed edges form an acyclic graph) with $(t-1)\cdot(\ell+1)$ auxiliary neurons also satisfying $\operatorname*{\mathrm{Pr}}[z^{t}=1]=p$ when given $X$ which is fixed for rounds $\{0,...,t\}$.

Proof.

Let $B=A\cup z$ – all non-input neurons. We simply produce $t-1$ duplicates of each auxiliary neuron $a\in A:\{a_{1},...,a_{t-1}\}$ and of $z:\{z_{1},...,z_{t-1}\}$, which are split into layers $B_{1},...,B_{t-1}$. For each incoming edge from a neuron $u$ to $v$ and each $i\geq 2$ we add an identical edge from $u_{i-1}$ to $v_{i}$. Any incoming edges from input neurons to $u$ are added to each $u_{i}$ for all $i\geq 1$. Finally connect $z$ to the appropriate neurons in $B_{t-1}$ (which may include $z_{t-1}$ if there is a self-loop in $\operatorname{\mathcal{N}}$).

In round $1$, the joint distribution of the spikes $B_{1}^{1}$ in $\mathcal{N}_{F}$ is identical to the distribution of $B^{1}$ in $\mathcal{N}$ since these neurons have identical incoming connections from the inputs, and since any incoming connections from other auxiliary neurons are not triggered in $\mathcal{N}$ since none of these neurons fire at time [math].

Assuming via induction that $B_{i}^{i}$ is identically distributed to $B^{i}$, since $B_{i+1}$ only has incoming connections from $B_{i}$ and the inputs which are fixed, then the distribution of $B_{i+1}^{i+1}$ identical to that of $B^{i+1}$. Thus $B_{t-1}^{t-1}$ is identically distributed to $B^{t-1}$, and since the output in $\mathcal{N}_{F}$ is only connected to $B_{t-1}$ its distribution is the same in round $t$ as in $\mathcal{N}$. ∎

Lemma 11** (Conversion to Distribution over Deterministic Threshold Circuits).**

Consider any spiking sigmoidal network $\mathcal{N}$ with $\ell$ auxiliary neurons, which given input $X\in\{0,1\}^{n}$ that is fixed for rounds $\{0,...,t\}$, has output neuron $z$ satisfying $\operatorname*{\mathrm{Pr}}[z^{t}=1]=p$. Then there is a distribution $\mathcal{D}$ over feedforward deterministic threshold circuits with $(t-1)\cdot(\ell+1)$ auxiliary gates that, for $\mathcal{N}_{D}\sim\mathcal{D}$ with output $z$, $\operatorname*{\mathrm{Pr}}_{\mathcal{D}}[z^{t}=1]=p$ when presented input $X$.

Proof.

We start with $\mathcal{N}_{F}$ obtained from Lemma 4.2. This circuit has $t-1$ layers of $\ell+1$ neurons $B_{1},...,B_{t-1}$. Given $X\in\{0,1\}^{n}$ that is fixed for rounds $\{0,...,t\}$, $\mathcal{N}_{F}$ has $\operatorname*{\mathrm{Pr}}[z^{t}=1]=p$, which matches the firing probability of the output $z$ in $\mathcal{N}$ in round $t$.

Let $\mathcal{D}$ be a distribution on deterministic threshold circuits that have identical edge weights to $\mathcal{N}_{F}$. Additionally, for any (non-input) neuron $u\in\mathcal{N}_{F}$, letting $\bar{u}$ be the corresponding neuron in the deterministic circuit, set the bias $b(\bar{u})=\eta$, where $\eta$ is distributed according to a logistic distribution with mean $\mu=b(u)$ and scale $s=\lambda$. The random bias is chosen independently for each $u$. It is well known that the cumulative density function of this distribution is equal to the sigmoid function. That is:

[TABLE]

Consider $\mathcal{N}_{D}\sim\mathcal{D}$ and any neuron $u$ in the first layer $B_{1}$ of $\mathcal{N}_{F}$. $u$ only has incoming edges from the input neurons $X$. Thus, its corresponding neuron $\bar{u}$ in $\mathcal{N}_{D}$ also only has incoming edges from the input neurons. Let $W=\sum_{x\in X}w(x,u)\cdot x^{0}$. Then we have:

[TABLE]

Let $\bar{B}_{i}$ denote the neurons in $\mathcal{N}_{D}$ corresponding to those in $B_{i}$. Since in round $1$, all neurons in $B_{1}$ fire independently and since all neurons in $\bar{B}_{1}$ fire independently as their random biases are chosen independently, the joint firing distribution of $B_{1}^{1}$ is identical to that of $\bar{B}_{1}^{1}$.

By induction assume that $\bar{B}_{i}^{i}$ is identically distributed (over the random choice of deterministic network $\mathcal{N}_{D}\sim\mathcal{D}$) to $B_{i}^{i}$. Then for any $u\in B_{i+1}$ we have by the same argument as above, conditioning on some fixed firing pattern $V$ of $B_{i}$ in round $i$:

[TABLE]

Conditioned on $B_{i}^{i}=V$, the neurons in $B_{i+1}$ fire independently in round $i+1$. So do the neurons of $\bar{B}_{i+1}$ due to their independent choices of random biases. Thus, the above implies that the distribution of $\bar{B}_{i+1}^{i+1}$ conditioned on $\bar{B}_{i}^{i}=V$ is identical to the distribution of $B_{i+1}^{i+1}$. This holds for all $V$, so, the full joint distribution of $\bar{B}_{i+1}^{i+1}$ is identical to that of ${B}_{i+1}^{i+1}$.

We conclude by noting that the same argument applies for the outputs of $\mathcal{N}_{F}$ and $\mathcal{N}_{D}$ since $\bar{B}_{t-1}^{t-1}$ is identically distributed to ${B}_{t-1}^{t-1}$. ∎

Lemma 11 is simple but powerful – it demonstrates the following:

The output distribution of a spiking sigmoid network is identical to the output distribution of a deterministic feedforward threshold circuit drawn from some distribution $\mathcal{D}$.

Thus, the performance of any spiking sigmoid network is equivalent to the performance of a randomized algorithm which first selects a linear threshold circuit using $\mathcal{D}$ and then applies this circuit to the input. This equivalence allows us to apply Yao’s minimax principal:

Lemma 12** (Application of Yao’s Principal).**

Assume there exists an SNN $\mathcal{N}$ with $\ell$ auxiliary neurons, which given any inputs $X\in\{0,1\}^{n}$ and $Y\in\{0,1\}^{\log n}$ which are fixed for rounds $\{0,...,t\}$, solves indexing with probability $\geq 1-\delta$ in $t$ rounds. Then there exists a feedforward deterministic linear threshold circuit $\mathcal{N}_{D}$ with $(t-1)\cdot(\ell+1)$ auxiliary gates which solves indexing with probability $\geq 1-\delta$ given $X,Y$ drawn uniformly at random.

Proof.

This follows from Yao’s principal [Yao77]. In short, given $X,Y$ drawn uniformly at random, $\mathcal{N}$ solves indexing with probability $\geq 1-\delta$ (since by assumption, it succeeds with this probability for any $X,Y$). By Lemma 11, $\mathcal{N}$ performs identically to an algorithm which selects a deterministic circuit from some distribution $\mathcal{D}$ and then applies it to the input. So at least one circuit in the support of $\mathcal{D}$ must succeed with probability $\geq 1-\delta$ on $X,Y$ drawn uniformly at random, since the success probability of $\mathcal{N}$ on the uniform distribution is just an average over the deterministic success probabilities. ∎

From Lemma 12 we have a corollary which concludes our reduction from our spiking sigmoid lower bound to a lower bound on deterministic indexing circuits.

Corollary 13** (Reduction to Deterministic Indexing on a Subset of Inputs).**

Assume there exists an SNN $\mathcal{N}$ with $\ell$ auxiliary neurons, which, given inputs $X\in\{0,1\}^{n}$ and $Y\in\{0,1\}^{\log n}$ which are fixed for rounds $\{0,...,t\}$, solves indexing with probability $\geq 1-\frac{1}{2n}$ in $t$ rounds. Then there exists some subset of inputs $\mathcal{X}\subseteq\{0,1\}^{n}$ with $|\mathcal{X}|\geq 2^{n-1}$ and a feedforward deterministic linear threshold circuit $\mathcal{N}_{D}$ with $(t-1)\cdot(\ell+1)$ auxiliary gates which solves indexing given any $X\in\mathcal{X}$ and any index $Y\in\{0,1\}^{\log n}$.

Proof.

Applying Lemma 12 yields $\mathcal{N}_{D}$ which solves indexing on uniformly random $X,Y$ with probability $1-\frac{1}{2n}$. Let $\mathbb{I}(X,Y)=1$ if $\mathcal{N}_{D}$ solves indexing correctly on $X,Y$ and [math] otherwise. Then:

[TABLE]

which in turn implies:

[TABLE]

If $\frac{1}{n}\sum_{Y\in\{0,1\}^{\log n}}(1-\mathbb{I}(X,Y))\neq 0$ then $\frac{1}{n}\sum_{Y\in\{0,1\}^{\log n}}(1-\mathbb{I}(X,Y))\geq\frac{1}{n}$ just by the fact that the sum is an integer. Thus, for (4) to hold, we must have $\frac{1}{n}\sum_{Y\in\{0,1\}^{\log n}}(1-\mathbb{I}(X,Y))=0$ for at least $\frac{1}{2}$ of the inputs $X\in\{0,1\}^{n}$. That is, $\mathcal{N}_{D}$ solves indexing for every input index on some subset $\mathcal{X}$ with $|\mathcal{X}|\geq\frac{1}{2}|\{0,1\}^{n}|\geq 2^{n-1}$. ∎

4.3 Lower Bound for Deterministic Indexing on a Subset of Inputs

With Corollary 13 in place, we now turn to lower bounding the size of a deterministic linear threshold circuit $\mathcal{N}_{D}$ which solves the indexing problem on some subset of inputs $\mathcal{X}$ with $|\mathcal{X}|\geq 2^{n-1}$. To do this, we employ VC dimension techniques first introduced for bounding the size of linear threshold circuits computing indexing on all inputs [Koi96].

Consider fixing some input $X\in\mathcal{X}$, such that the output of $\mathcal{N}_{D}$ is just a function of the index $Y$. Specifically, with $X$ fixed, $\mathcal{N}_{D}$ computes the function $f_{X}:\{0,1\}^{\log n}\rightarrow\{0,1\}$ whose truth table is given by $X$. Note that the output of $\mathcal{N}_{D}$ with $X$ fixed is equivalent to the output of a feedforward linear threshold circuit $\mathcal{N}_{D}^{X}$ where each gate with an incoming edge from $x_{i}\in X$ has its threshold adjusting to reflect the weight of this edge if $x_{i}=1$.

We define two sets of functions. Let $\mathcal{F}=\{f_{X}|X\in\mathcal{X}\}$ be all functions computable using some $\mathcal{N}_{D}^{X}$ as defined above. Further, let $\mathcal{G}$ be the set of all functions computabled by any circuit $\mathcal{N}_{D}^{\prime}$ which is generated by removing the input gates of $\mathcal{N}_{D}$ and adjusting the threshold on each remaining gate to reflect the effects of any inputs with $x_{i}=1$. We have $\mathcal{F}\subseteq\mathcal{G}$ and hence, letting $VC(\cdot)$ denote the VC dimension of a set of functions have: $VC(\mathcal{F})\leq VC(\mathcal{G}).$ We can now apply two results. The first gives a lower bound $VC(\mathcal{F})$:

Lemma 14** (Corollary 3.8 of [AB09] – Consequence of Sauer’s Lemma [Sau72, She72]).**

For any set of boolean functions $\mathcal{H}=\{h\}$ with $h:\{0,1\}^{\log n}\rightarrow\{0,1\}$:

[TABLE]

We next upper bound $VC(\mathcal{G})$. We have the following, whose proof is in Appendix B:

Lemma 15** (Linear Threshold Circuit VC Bound).**

Let $\mathcal{H}$ be the set of all functions computed by a fixed feedforward linear threshold circuit with $m\geq 2$ gates (i.e. fixed edges and weights), where each gate has a variable threshold. Then: $VC(\mathcal{H})\leq 3m\log m.$

Applying the bounds of Lemmas 14 and 15 along with $VC(\mathcal{F})\leq VC(\mathcal{G})$ gives:

Lemma 16** (Deterministic Circuit Lower Bound).**

For any set $\mathcal{X}\subseteq\{0,1\}^{n}$ with $|\mathcal{X}|\geq 2^{n-1}$, any feedforward deterministic linear threshold circuit $\mathcal{N}_{D}$ with $m$ non-input gates which solves indexing given any $X\in\mathcal{X}$ and any index $Y\in\{0,1\}^{\log n}$ must have $m=\Omega\left(\frac{n}{\log^{2}n}\right).$

Proof.

Let $\mathcal{F}$ and $\mathcal{G}$ be as defined in the beginning of the section. We have $VC(\mathcal{F})\leq VC(\mathcal{G})$. At the same time, by Lemma 14 we have

[TABLE]

for some fixed constant $c$. By Lemma 15 we have $VC(\mathcal{G})\leq 3m\log m.$ We thus can conclude that $\frac{cn}{\log n}\leq 3m\log m$, and so $m=\Omega\left(\frac{n}{\log^{2}n}\right).$ ∎

We conclude by proving our main lower bound:

Proof of Theorem 8.

The existence of a spiking sigmoidal network with $\ell$ auxiliary neurons, solving indexing with probability $\geq 1-\frac{1}{2n}$ in $t$ rounds implies via Corollary 13 the existence of a feedforward deterministic linear threshold circuit with $(t-1)\ell+1$ non-input gates solving indexing on some subset of inputs $\mathcal{X}$ with $|\mathcal{X}|\geq 2^{n-1}$. Thus by Lemma 16 we must have $\ell\cdot t=\Omega\left(\frac{n}{\log^{2}n}\right)$. ∎

5 Applications to Similarity Testing and Compression

5.1 Similarity Testing

Theorem 17** (Similarity Testing).**

There exists an SNN with $O\left(\frac{\sqrt{n}\log n}{\epsilon}\right)$ auxilary neurons that solves the approximate equality testing problem in $O(\sqrt{n})$ rounds. Specifically, given inputs $X_{1},X_{2}\in\{0,1\}^{n}$ which are fixed for all rounds $t\in\{0,..,5\sqrt{n}+2\}$, the output $z$ satisfies w.h.p. $z^{5\sqrt{n}+2}=1$ if $\operatorname{d_{H}}(X_{1},X_{2})\geq\epsilon n$. Further if $X_{1}=X_{2}$ then $z^{5\sqrt{n}+2}=0$ w.h.p.

Our similarity testing network will use $K=\Theta\left(\frac{\log n}{\epsilon}\right)$ copies of our Neuro-RAM from Theorem 6, labeled $S_{1,k}$ and $S_{2,k}$ for all $k\in\{1,...,K\}$. The idea will be to employ $\log n$ auxiliary neurons $Y_{k}=y_{1,k},...,y_{\log n,k}$ whose values encode a random index $i\in\{0,...,n-1\}$. By feeding the inputs $(X_{1},Y_{k})$ and $(X_{2},Y_{k})$ into $S_{1}$ and $S_{2}$, we can check whether $X_{1}$ and $X_{2}$ match at position $i$. Checking $\Theta\left(\frac{\log n}{\epsilon}\right)$ different random indices suffices that identify if $\operatorname{d_{H}}(X_{1},X_{0})\geq\epsilon n$ w.h.p. Additionally, if $X_{1}=X_{0}$, they will never differ at any of the checks, and so the output will never be triggered. We use the following:

Observation 18**.**

Consider $X_{1},X_{2}\in\{0,1\}^{n}$ with $\operatorname{d_{H}}(X_{1},X_{0})\geq\epsilon n$. Let $i_{1},...,i_{T}$ be chosen independently and uniformly at random in $\{0,...,n-1\}$. Then for $T=\frac{c\ln n}{\epsilon}$,

[TABLE]

Proof.

For any fixed $t$, $\operatorname*{\mathrm{Pr}}[(X_{1})_{i_{t}}=(X_{2})_{i_{t}}]=1-\frac{\epsilon n}{n}=1-\epsilon$ as we select indices at random. Additionally, each of these events is independent since $i_{1},...i_{T}$ are chosen independently so: $\operatorname*{\mathrm{Pr}}[(X_{1})_{i_{t}}=(X_{2})_{i_{t}}\text{ for all }t\in 1,...,T]\leq(1-\epsilon)^{T}=\left(1-\epsilon)^{1/\epsilon}\right)^{c\ln n}\leq\frac{1}{e^{c\ln n}}\leq\frac{1}{n^{c}}.$ ∎

5.1.1 Implementation Sketch

It is clear that the above strategy can be implemented in the spiking sigmoidal network model – we sketch the construction here. By Theorem 6, we require $O\left(\frac{\sqrt{n}\log n}{\epsilon}\right)$ auxiliary neurons for the $2K=\Theta\left(\frac{\log n}{\epsilon}\right)$ neuro-RAMs employed, which dominates all other costs.

It suffices to present a random index to each pair of neuro-RAMs $S_{1},k$ an $S_{2},k$ for $5\sqrt{n}$ rounds (the number of rounds required for the network of Theorem 6 to process an $n$-bit input). To implement this strategy, we need two simple mechanisms, described below.

Random Index Generation:

For each of the $\log n$ index neurons in $Y_{k}$ we set $\operatorname{b}(y_{i})=0$ and add a self-loop $w(y_{i},y_{i})=2$. In round $1$, since they have no-inputs, each neuron has potential [math] and fires with probability $1/2$. Thus, $Y_{k}^{1}$ represents a random index in $\{0,...,n-1\}$. To propagate this index we can use a single auxiliary inhibitory neuron $g$, which has bias $\operatorname{b}(g)=1$ and $w(x,g)=2$ for every input neuron $x$. Thus, $g$ fires w.h.p. in round $1$ and continues firing in all later rounds, as long as at least one input fires.

We add an inhibitory edge from $g$ to $y_{i}$ for all $i$ with weight $w(g,y_{i})=-1$. The inhibitory edges from $g$ will keep the random index ‘locked’ in place. The inhibitory weight of $-1$ prevents any $y_{i}$ without an active self-loop from firing w.h.p. but allows any $y_{i}$ with an active self-loop to fire w.h.p. since it will still have potential $\operatorname{b}(y_{i})+w(y_{i},y_{i})-1=1$.

If both inputs are [math], $g$ will not fire w.h.p. However, here our network can just output [math] since $X_{1}=X_{2}$ so it does not matter if the random indices stay fixed.

Comparing Outputs:

We next handle comparing the outputs of $S_{1,k}$ and $S_{2,k}$ to perform equality checking. We use two auxiliary neurons – $f_{1,k}$ and $f_{2,k}$. $f_{1,k}$ is excitatory and fires w.h.p. as long as long as at least one of $S_{1,k}$ or $S_{2,k}$ has an active output. $f_{2,k}$ is an inhibitor that fires only if both $S_{1,k}$ and $S_{2,k}$ have active outputs. We then connect $f_{1,k}$ to our output $z$ with weight $w(f_{1,k},z)=2$ and connect $f_{2,k}$ with weight $w(f_{2},z)=-2$ for all $k$. We set $\operatorname{b}(z)=1$. In this way, $z$ fires in round $5\sqrt{n}+2$ w.h.p. if for some $k$, exactly one of $S_{1,k}$ or $S_{2,k}$ has an active output in round $5\sqrt{n}$ and hence an inequality is detected. Otherwise, $z$ does not fire w.h.p. This behavior gives the output condition of Theorem 17.

5.2 Randomized Compression

We conclude by discussing informally how our neuro-RAM can be applied beyond similarity testing to other randomized compression schemes. Consider the setting where we are given $n$ input vectors $X_{i}\in\{0,1\}^{d}$. Let $\mathbf{X}\in\{0,1\}^{n\times d}$ denote the matrix of all inputs. Think of $d$ as being a large ambient dimension, which we would like to reduce before further processing.

One popular technique is Johnson-Lindenstrauss (JL) random projection, where $\mathbf{X}$ is multiplied by a random matrix $\boldsymbol{\Pi}\in\mathbb{R}^{d\times d^{\prime}}$ with $d^{\prime}<<d$ to give the compressed dataset $\mathbf{\tilde{X}}=\mathbf{X}\boldsymbol{\Pi}$. Regardless of the initial dimension $d$, if $d^{\prime}$ is set large enough, $\mathbf{\tilde{X}}$ preserves significant information about $\mathbf{X}$. $d^{\prime}=\tilde{O}(\log n)$ is enough to preserve the distances between all points w.h.p. [KN14], $d^{\prime}=\tilde{O}(k)$ is enough to use $\mathbf{\tilde{X}}$ for approximate $k$-means clustering or $k$-rank approximation [BZD10, CEM*+*15], and $d^{\prime}=\tilde{O}(n)$ preserves the full covariance matrix of the input and so $\mathbf{\tilde{X}}$ can be used for approximate regression and many other problems [CW13, Sar06].

JL projection has been suggested as a method for neural dimensionality reduction [AZGMS14, GS12], where $\boldsymbol{\Pi}$ is viewed as a matrix of random synapse weights, which connect the input neurons representing $\mathbf{X}$ to the output neurons representing $\mathbf{\tilde{X}}$. While this view is quite natural, we often want to draw $\boldsymbol{\Pi}$ with fresh randomness for each input $\mathbf{X}$. This is not possible using changing synapse weights, which evolve over a relatively long time scale. Fortunately, it is possible to simulate these random connections using our neuro-RAM module.

Typically, $\boldsymbol{\Pi}$ is sparse so that it can be multiplied by efficiently. In one of the most efficient constructions [CW13], it has just a single nonzero entry in each row which is chosen randomly to be $\pm 1$ and placed in a uniform random position in the row. Thus, computing a single bit of $\mathbf{\tilde{X}}=\mathbf{X}\boldsymbol{\Pi}$ requires selecting on average $d/d^{\prime}$ random columns of $\mathbf{X}$, multiplying their entries by a random sign and summing them together. This can be done with a set of neuro-RAMS, each using $O(\sqrt{d})$ auxiliary neurons which select the random columns of $\mathbf{X}$. In total we will needs $\tilde{O}(d/d^{\prime})$ networks – the maximum column sparsity of $\boldsymbol{\Pi}$ with high probability, yielding $O(d^{3/2}/d^{\prime})$ auxiliary neurons total. In contrast, a naive simulation of random edges using spiking neurons would require $\Theta(d)$ auxiliary neurons, which is less efficient whenever $d^{\prime}>d^{3/2}$. Additionally, our neuro-RAMs can be reused to compute multiple entries of $\mathbf{\tilde{X}}$, which is not the case for the naive simulation.

Traditionally, the value of an entry of $\mathbf{\tilde{X}}$ is a real number, which cannot be directly represented in a spiking neural network. In our construction, the value of the entry is encoded in its potential, and we leave as an interesting open question how this potential should be decoded or otherwise used in downstream applications of the compression.

Acknowledgments

We are grateful to Mohsen Ghaffari. Some of the ideas of this paper came up while visiting him at ETH. We would like to thank Sergio Rajsbaum, Ron Rothblum and Nir Shavit for helpful discussions.

Appendix A Missing Details for Implementing Neuro-RAM

A.1 First Stage: Bucket Selection

To implement the bucket selection stage, we connect the neurons in $Y_{1}$ to each $e_{i}$ such that if $i=\operatorname{dec}(Y_{1})$ the potential of $e_{i}$ is increased significantly, and if $i\neq\operatorname{dec}(Y_{1})$ the potential of $e_{i}$ remains the same. By setting this potential increase to a very large value and the bias $\operatorname{b}(e_{i})$ to a correspondingly very large value, $e_{i}$ will not fire with high probability unless $i=\operatorname{dec}(Y_{1})$. This selection phase can be implemented with $O(\sqrt{n})$ auxilary neurons.

Specifically, for each $y_{j}\in Y_{1}$, we have two neurons $y_{j,1}$, $y_{j,1^{\prime}}$ connected as follows:

• •

$y_{j,1}$ is an excitatory neuron with $\operatorname{b}(y_{j,1})=1$ and $w(y_{j},y_{j,1})=2$. In this way, if $y_{j}^{t}=1$, $\operatorname{pot}(y_{j,1},t)=1$ and so $y_{i,1}^{t+1}=1$ w.h.p.

• •

$y_{j,1^{\prime}}$ is an inhibitory neuron with $\operatorname{b}(y_{j,1^{\prime}})=1$ and $w(y_{j},y_{j,1^{\prime}})=2$. So again, if $y_{j}^{t}=1$, $\operatorname{pot}(y_{j,1^{\prime}},t)=1$ and so $y_{j,1}^{t+1}=1$ w.h.p.

The behavior of $y_{j,1}$ and $y_{j,1^{\prime}}$ can be summarized as:

Lemma 19**.**

For any $t$, if $y_{j}^{t}=1$ then w.h.p. $y_{j,1}=y_{j,1^{\prime}}=1$.

For each $e_{i}$, we then have an auxiliary excitatory neuron $g_{i}$. Connected as follows:

• •

$w(y_{j,1},g_{i})=2$ if $\operatorname{bin}(i)_{j}=1$ and [math] otherwise.

• •

$w(y_{j,1^{\prime}},g_{i})=-2$ if $\operatorname{bin}(i)_{j}=0$ and [math] otherwise.

• •

$\operatorname{b}(g_{i})=2\|\operatorname{bin}(i)\|_{1}-1$.

• •

$w(g_{i},e_{i})=2^{\sqrt{n}+2}$.

We have the following lemma:

Lemma 20**.**

For any $t$, if $Y_{1}^{t}=\operatorname{bin}(i)$ then w.h.p. $g_{i}^{t+2}=1$ and $g_{k}^{t+2}=0$ for all $k\neq i$.

Proof.

The connections to $y_{j,1}$ and $y_{j,1^{\prime}}$ and Lemma 19 ensure that if $Y_{1}^{t}=\operatorname{bin}(i)$, w.h.p.

[TABLE]

Thus, if $Y_{1}^{t}=\operatorname{bin}(i)$, $g_{i}^{t+2}=1$ w.h.p. Similarly, if $Y_{1}^{t}=\operatorname{bin}(i)$ then for any $k\neq i$:

[TABLE]

where the bound that the sum is $\leq 2\|\operatorname{bin}(k)\|_{1}-2$ follows from the fact that $\operatorname{bin}(i)_{j}\neq\operatorname{bin}(k)_{j}$ for some $j$. Thus $g_{k}^{t+2}=0$ w.h.p. completing the lemma by a union bound over all $g_{k}$. ∎

By Lemma 20 we can ensure that $e_{i}$ only fires if it corresponds to the bucket selected by $Y$ – i.e., if $Y_{1}=\operatorname{bin}(i)$. To do this we need one more fact, which will be clear after defining our full network:

Fact 21**.**

The total weight of incoming excitatory connections to $e_{i}$ for any $i$, excluding the connection from $g_{i}$, is upper bounded by $2^{\sqrt{n}+2}$.

Lemma 22** (Bucket Selection).**

For any $t$, if $Y_{1}^{t}=\operatorname{bin}(i)$ then w.h.p. $e_{k}^{t+3}=0$ for all $k\neq i$.

Proof.

By the setting of $\operatorname{b}(e_{k})=2^{\sqrt{n}+2}+2^{\sqrt{n}}-1$, Fact 21, and Lemma 20 w.h.p.

[TABLE]

so w.h.p. $e_{k}^{t+3}=0$, giving the lemma after union bounding over all $e_{k}$. ∎

We note that if $Y_{1}^{t}=\operatorname{bin}(i)$ then $e_{i}$ will receive a potential of $2^{\sqrt{n}+2}$ from $g_{i}$ and so its effective bias will shift to $2^{\sqrt{n}}+1$, which as we will see, will be appropriate for the remainder of the algorithm.

A.2 Second Stage: Bucket Decoding

We now discuss the decoding phase of our network.

A.2.1 Clock Mechanism

For this phase we need a clock mechanism. Specifically, we have some initiator neuron $c_{0}$, with $w(x_{i},c_{0})=2$ for all inputs $x_{i}$ and $\operatorname{b}(c_{0})=1$. Thus, $c_{0}$ fires w.h.p. in response to at least one input firing. We then have $5\sqrt{n}$ additional excitatory neurons $c_{1},...,c_{5\sqrt{n}}$ and $5\sqrt{n}$ additional inhibitory neurons $c_{1^{\prime}},...,c_{5\sqrt{n}^{\prime}}$. We set $\operatorname{b}(c_{i})=\operatorname{b}(c_{i^{\prime}})=1$ and $w(c_{i-1},c_{i})=w(c_{i-1},c_{i}^{\prime})=2$. Further, we set $w(c_{1}^{\prime},c_{1})=-2$, and for all $i<5\sqrt{n}$, $w(c_{i}^{\prime},c_{0})=-2n$. This gives the following clocking property:

Lemma 23** (Clock Mechanism).**

If $c_{0}^{t}=1$ for some round $t$, then for all $i\leq 5\sqrt{n}$, $c_{i}^{t+i}=1$. Further, for all $j\leq 5\sqrt{n}$, with $j\neq i$ $c_{i}^{t+j}=0$ w.h.p.

We place inhibitory connections from each $c_{i}^{\prime}$ back to $c_{0}$, which prevent the clock from ‘restarting’ before it has finished a complete cycle. We do not connect the last inhibitor $c_{5\sqrt{n}}$ to $c_{0}$ (in fact, this inhibitor can be removed from the network). This ensures that once the $c_{5\sqrt{n}}$ fires, in the next round, as long as at least one input is active, $c_{0}$ will fire w.h.p. restarting the clock. While this is not necessary for the correctness of our neuro-RAM, it will be useful in applications that reuse this network to process multiple inputs.

Proof.

Since $c_{0}^{t}=1$, $\operatorname{pot}(c_{1},t)=\operatorname{pot}(c_{1^{\prime}},t)=-1+2=1$ and so $c_{1}^{t+1}=c_{1^{\prime}}^{t+1}=1$ w.h.p. Thus, due to the inhibitory connection from $c_{1^{\prime}}$, and since $c_{0}$ has a total of $2n$ excitatory weight from all $n$ inputs, w.h.p. $\operatorname{pot}(c_{0},t+1)\leq-1+2n-2n\leq-1$ so $c_{0}^{t+2}=0$ w.h.p. Finally, due to the inhibitory connection from $c_{1^{\prime}}$ to $c_{1}$, w.h.p. $\operatorname{pot}(c_{1},t+1)\leq-1+2-2\leq-1$ so $c_{1}^{t+2}=0$ w.h.p.

Now for any $i>1$ assume by induction that for $j<i$, $c_{j}^{t+j}=1$, $c_{j}^{t+j+1}=0$ and $c_{0}^{t+j+1}$ = 0. This implies that $\operatorname{pot}(c_{i},t+i-1)=\operatorname{pot}(c_{i^{\prime}},t+i-1)=1$ and thus $c_{i}^{t+i}=c_{i^{\prime}}^{t+i}=1$ w.h.p. Additionally, since $c_{i-1}^{t+i}=0$, $\operatorname{pot}(c_{i},t+i)=-1$ and so $c_{i}^{t+i+1}=0$ w.h.p. Finally, since $c_{i^{\prime}}^{t+i}=1$ w.h.p. $\operatorname{pot}(c_{0},t+i)\leq-1+2n-2n\leq-1$ and so $c_{0}^{t+i+1}=0$ w.h.p. Thus, the inductive assumption is fulfilled up to round $i$.

The lemma follows simply by noting that since $c_{0}$ does not fire in any round after round $t+1$, it never excites $c_{1}$ and so later neurons in the chain are not re-excited and at round $t+i$, $c_{j}^{t+i}=0$ w.h.p. for all $j\leq i-1$. We union bound over all failure events and have the full result w.h.p. ∎

We finally state the following immediate corollary of Lemma 23 which will be useful:

Corollary 24**.**

If any input in $X$ fires at time $t$ then w.h.p. for all $i$, $c_{i}^{i+1}=1$ and for all $j\leq 5\sqrt{n}$ with $j\neq i$, $c_{i}^{j+1}=0$.

Proof.

This just follows from the fact that if any input fires in round $t$, $c_{0}$ fires w.h.p. in round $t+1$. We then have the result by Lemma 23. ∎

A.2.2 Successive Decoding

With the clock mechanism of Lemma 23 in place we can give our decoding algorithm. As discussed, we have three decoding neurons for each bit, labeled $d_{j,1}$, $d_{j,2}$, and $d_{j,3}$. We describe the specific setups for each below.

Output Triggering

$d_{j,1}$ is responsible for triggering the output $z$ to fire if bit $j$ of $X_{\operatorname{dec}(Y_{1})}$ is $1$. We set $w(e_{i},d_{j,1})=2$ for all $i,j$. We additionally set $w(d_{j,1},z)=2$ for all $j$ and $\operatorname{b}(z)=1$, such that whenever one of these triggering neurons fires, the output $z$ fires. We also add a self-loop with weight $w(z,z)=2$ such that once the output fires, it continues to fire w.h.p.

Like $e_{i}$, $d_{j,1}$ is equipped with an auxilary neuron $f_{j}$ which fires w.h.p. iff $Y_{2}=\operatorname{bin}(j)$. Specifically, we connect $f_{j}$ to $Y_{2}$ in an identical manner to how we connected $g_{j}$ to $Y_{1}$ and have the following analog of Lemma 20:

Lemma 25**.**

For any $t$, if $Y_{2}^{t}=\operatorname{bin}(j)$ then w.h.p. $f_{j}^{t+2}=1$ and $f_{k}^{t+2}=0$ for all $k\neq j$.

We set $w(y_{j},d_{j,1})=2$. Additionally, to insure that $d_{j,1}$ only fires at the appropriate step of decoding we connect it to our clock mechanism. We set $w(c_{l},d_{j,1})=2\sqrt{n}$ for $l=5j+2$. Finally we set $\operatorname{b}(d_{j,1})=2\sqrt{n}+3$. This gives the following lemma:

Lemma 26**.**

Assume that the inputs $X$, $Y_{1}$, and $Y_{2}$ remain fixed for $t\in\{0,...,5\sqrt{n}\}$ and that $Y_{1}=\operatorname{bin}(i)$ and $Y_{2}=\operatorname{bin}(j)$. Then $d_{j,1}^{t}=1$ for $t=5j+4$ w.h.p. if $e_{i}^{t}=1$ for $t=5j+3$. Otherwise, w.h.p. $d_{j,1}^{t}=0$ for all $t\leq 5\sqrt{n}$.

Proof.

$d_{j,1}$ will not fire w.h.p. in round $t$ unless $c_{l}$ (for $l=5j+2$) fires in round $t-1$. This is because the weight of all incoming connections to $d_{j,1}$ from the $\sqrt{n}$ encoding neurons and $y_{j}$ is $2\sqrt{n}+2$ which is not enough to overcome the bias $\operatorname{b}(d_{j,1})=2\sqrt{n}+3$.

Additionally, assuming $X^{t}\neq 0$, then at least one input fires in round [math], so by Corollary 24, $c_{l}$ fires w.h.p. in round $5j+3$ and in no other rounds by Lemma 23. So if $d_{j,1}$ fires, w.h.p. it must be in round $t=5j+4$.

Note that to have $d_{j,1}^{t}=1$ w.h.p. we must additionally have $f_{j}^{t-1}=1$ and $e_{\operatorname{dec}(Y_{1})}^{t-1}=1$. Otherwise, since by Lemma 22, $e_{\operatorname{dec}(Y_{1})}$ is the only encoding neuron that fires after round $3$:

[TABLE]

and so $d_{j,1}^{t}=0$ w.h.p. By Lemma 25, if $Y_{2}^{t^{\prime}}=\operatorname{bin}(j)$ for all $t^{\prime}\in\{0,...,5\sqrt{n}\}$ then $y_{j}$ will fire in all rounds after round $2$ and hence in round $t-1$. Thus $d_{1,j}$ will fire w.h.p. if $e_{\operatorname{dec}{Y_{1}}}$ fires in $t-1$ as well. If $Y_{2}\neq\operatorname{bin}(j)$ or if $e_{\operatorname{dec}{Y_{1}}}$ does not fire in this round, then $d_{1,j}$ will not fire with high probability, giving the lemma. We conclude by noting that we assumed that $X^{t}\neq 0$. If $X^{t}=0$, then $c_{0}$ will never be triggered, and thus no $d_{j,1}$ will ever fire, so $z$ will never fire. This is a correct output, as all bits of the input are [math]. ∎

From Lemma 26 we have the following simple corollary:

Corollary 27**.**

Assume that the inputs $X$, $Y_{1}$, and $Y_{2}$ remain fixed for $t\in\{0,...,5\sqrt{n}\}$ and that $Y_{1}=\operatorname{bin}(i)$ and $Y_{2}=\operatorname{bin}(j)$. Then $z^{5\sqrt{n}}=1$ w.h.p. if $e_{i}^{t}=1$ for $t=5j+3$. $z^{5\sqrt{n}}=0$ w.h.p. otherwise.

Proof.

This follows directly from Lemma 26 and the fact that we set $\operatorname{b}(z)=1$ and $w(d_{j,1},z)=2$ for all $j\in 1,...,\sqrt{n}$. Additionally, once $z$ fires, it continues firing as we added a self-loop with weight $2$. Thus it fires w.h.p. in round $5\sqrt{n}$. ∎

Potential Reading

Corollary 27 shows that as long as $\operatorname{bin}(i)=Y_{1}$, $\operatorname{bin}(j)=Y_{2}$, and $X_{\sqrt{n}i+j}=1$ causes $e_{i}$ to fire at round $5j+3$ then $z$ will fire. If $e_{i}$ does not fire in this round then $z$ will fire in a round w.h.p. iff $X_{\operatorname{dec}(Y)}=1$. Thus is remains to demonstrate how to ensure that $e_{i}$ fires in this round if $X_{\sqrt{n}i+j}=1$ and does not fire if $X_{\sqrt{n}i+j}=0$.

To do this we use an excitatory neurons $d_{j,2},d_{j,3^{\prime}}$ and the inhibitory neuron $d_{j,3}$. We set $\operatorname{b}(d_{j,2})=1$ and $\operatorname{b}(d_{j,3})=\operatorname{b}(d_{j,3^{\prime}})=3$. We set $w(e_{i},d_{j,3})=w(e_{i},d_{j,3^{\prime}})=2$ for all $i,j\in\{0,...,\sqrt{n}-1\}$ and finally $w(c_{l},d_{j,2})=2$ for $l=5j+2$ and $w(c_{l},d_{j,3})=2$ for $l=5j+2$. Finally we create self loops $w(d_{j,2},d_{j,2})=2$ and $w(d_{j,3^{\prime}},d_{j,3^{\prime}})=4$. and $w(d_{j,3^{\prime}},d_{j,3})=4$. This gives the following lemmas:

Lemma 28**.**

Assume that $X^{0}\neq 0$ (i.e. at least one input fires in round [math]). With high probability, for every $j$, $d_{j,2}^{t}=0$ for all $t<5j+4$ and $d_{j,2}^{t}=1$ for all $5j+4\leq t\leq 5\sqrt{n}$.

Proof.

We have $\operatorname{b}(d_{j,2})=1$, and $w(c_{l},d_{j,2})=2$ so as long as $X^{t}\neq 0$, by Corollary 24, $c_{l}$ fires in round $5j+3$, $d_{j,2}$ will fire in the next round w.h.p. Further, it will continue to fire in all successive rounds w.h.p. due to its self-loop with $w(d_{j,2},d_{j,2})=2$. Since it does not fire in round [math], its self-loop will be inactive and it will not fire in any round before $5j+4$. ∎

Lemma 29**.**

Assume that $X$ and $Y_{1}$ remain fixed for $t\in\{0,...,5\sqrt{n}\}$, that $X\neq 0$, that $Y_{1}=\operatorname{bin}(i)$. For every $j$, if $e_{i}^{t}=1$ for $t=5j+3$, then w.h.p. $d_{j,3}^{t}=1$ for all $t\geq 5j+4$. Otherwise, w.h.p. $d_{j,3}^{t}=0$ for all $t\leq 5\sqrt{n}$.

Proof.

Since $\operatorname{b}(d_{j,3})=3$, in order of $d_{j,3}$ to fire in round $t$ if it did not fire in round $t-1$ (and hence $d_{j,3^{\prime}}$, which has identical connections, was not activated w.h.p. ) we must have $e_{i}^{t-1}=1$ for some $i$ and $c_{l}^{t-1}=1$ for $l=5j+2$. By Corollary 24, since $X^{0}\neq 0$, $c_{l}$ fires w.h.p. in round $5j+3$ and no other round, so if $d_{j,3}$ fires it must be in round $5j+4$ w.h.p. Since only $e_{\operatorname{dec}(Y_{1})}$ fires in any round after round $3$ w.h.p. by Lemma 22, we must have $e_{\operatorname{dec}(Y_{1})}^{5j+3}=1$ in order for $d_{j,3}$ to fire. We finally note that once $d_{j,3}$ fires, it will continue firing w.h.p. in each round due to $d_{j,3^{\prime}}$ whose excitatory connection excites it (the connection from $d_{j,3^{\prime}}$ behaves as an excitatory self-loop, which $d_{j,3}$ is not allowed to have directly since it is an inhibitor). Further, it does not fire in any round before $t=5j+4$ since it does not fire in round [math] so $d_{j,3^{\prime}}$ will be inactive. ∎

We now discuss how $d_{j,2}$ and $d_{j,3}$ provide feedback to $e_{i}$. We set for all $i$,

[TABLE]

We can verify Fact 21: The total weight on each $e_{i}$ from the $d_{j,2}$ neurons is at most $\sum_{j=0}^{\sqrt{n}-1}2^{\sqrt{n}-j-1}\leq 2^{\sqrt{n}}$. The total weight from the inputs is at most $\sum_{j=0}^{\sqrt{n}-1}2^{\sqrt{n}-j}\leq 2^{\sqrt{n}+1}$, so overall the total excitatory weight is at most $2^{\sqrt{n}+2}$. We can finally prove:

Lemma 30**.**

Assume that $X$, $Y_{1}$ remain fixed for $t\in\{0,...,5\sqrt{n}\}$ and that $Y_{1}=\operatorname{bin}(i)$. For all $k,j$, $e_{k}^{5j+3}=1$ w.h.p. if $X_{k\sqrt{n}+j}=1$ and $k=i$. Otherwise, $e_{k}^{5j+3}=0$ w.h.p.

Proof.

We first note that if $X=0$, then no $e_{k}$ will ever fire and the lemma will be correct trivially. So we assume $X\neq 0$. By Lemma 22, for $k\neq i$, $e_{k}^{t}$ for all $3\leq t\leq 5\sqrt{n}$ w.h.p. which immediately gives the result in this case. So now consider $k=i$. For $j=1$, since we set $\operatorname{b}(e_{i})=2^{\sqrt{n}+2}+2^{\sqrt{n}}-1$, since $Y_{1}^{t}=\operatorname{bin}(i)$ for $t=0$, by Lemma 20, $g_{i}$ fires in round $2$. Thus w.h.p.

[TABLE]

where the last step follows since $w(g_{i},e_{i})=2^{\sqrt{n}+2}$ and since neither $d_{j,2}$ nor $d_{j,3}$ fire w.h.p. before round $4$ (see Lemmas 28 and 29). Now, $\sum_{j=0}^{\sqrt{n}-1}X_{i\sqrt{n}+j}\cdot 2^{\sqrt{n}-j}\geq 2^{\sqrt{n}}$ if $X_{i\sqrt{n}}=1$ and $\sum_{j=0}^{\sqrt{n}-1}X_{i\sqrt{n}+j}\cdot 2^{\sqrt{n}-j}\leq 2^{\sqrt{n}}-2$ otherwise. Thus $e_{i}$ fires w.h.p. in round $3$ if $X_{i\sqrt{n}}=1$ and does not fire w.h.p. otherwise. This completes the lemma in the base case of $j=0$.

Now consider $j\geq 1$ and assume the lemma holds w.h.p. for all $j^{\prime}<j$. By this inductive assumption and Lemmas 28 and 29 at round $t=5j+2$ we have w.h.p.

[TABLE]

And again, this potential is $\geq 1$ if $X_{i\sqrt{n}+j}=1$ and $\leq-1$ otherwise. Thus, $e_{i}^{5j+3}=1$ w.h.p. if $X_{i\sqrt{n}+j}=1$ and is [math] otherwise w.h.p. This completes the lemma. ∎

With this lemma in hand, we can now prove our main result Theorem 6.

Proof of Theorem 6.

The theorem follows from Corollary 27 combined with Lemma 30. ∎

Proof of Observation 7.

To run the indexing algorithm multiple times, the network must ‘reset’ to a state in which all auxiliary neurons do not fire for a round. This is notably important for $d_{j,2}$, $d_{j,3^{\prime}}$, and $z$. Once these neurons spike, they propagate the spike via a self-loop and will fire continuously w.h.p. unless they receive some external inhibition. A simple way to achieve a reset is to add an inhibitory neuron $r$ with $\operatorname{b}(r)=1$ and $w(c_{5\sqrt{n}-2},r)=2$. Thus, since $c_{5\sqrt{n}-2}$ fires w.h.p. in round $5\sqrt{n}-1$, $r$ will fire w.h.p. in round $5\sqrt{n}$. We can add an inhibitory synapse from $r$ to each decoding neuron and to $z$, with arbitrarily large weight. In this way, in round $5\sqrt{n}+1$, none of these neurons will fire with high probability, and the computation will proceed as described. Round $5\sqrt{n}+1$ can be identified with round [math] in the statement of Theorem 6. ∎

Appendix B Missing Proofs for the Lower Bound

Proof of Lemma 15:

We first define a generalization of VC dimension, which measures the number of dichotomies that a class of functions can induce over a set.

Definition 31**.**

For a class of functions $\mathcal{H}:X\rightarrow\{0,1\}$, let $\Delta_{\mathcal{H}}(z)$ be the maximum over sets $A\subseteq X$ with $|A|=z$ of $\Delta_{\mathcal{H}}(A)$, the number of different partitions of $A$ that can be induced by some $h\in\mathcal{H}$. $VC(\mathcal{H})$ is the maximum $z$ with $\Delta_{\mathcal{H}}(z)=2^{z}$.

Lemma 32** (Theorem 1 of [BH89]).**

Let $\mathcal{H}$ be the class of functions computed by a fixed feedforward architecture with $m$ nodes, where each node $v_{i}$ can be chosen to compute any function in the class $\mathcal{H}_{i}$. Then $\Delta_{\mathcal{H}}(z)\leq\prod_{i=1}^{m}\Delta_{\mathcal{H}_{i}}(z)$.

Applying Lemma 32 to a fixed threshold circuit with programmable thresholds gives Lemma 15. For a threshold gate with fixed input weights and a variable threshold, we trivially have $\Delta_{\mathcal{H}_{i}}(z)\leq z$, since for $z$ inputs, there are $z$ possible functions that can computed by varying the threshold of the circuit (since edge weights are fixed, any input is just mapped to a single real number which is thresholded). We thus have $\Delta_{\mathcal{H}}(z)\leq z^{m}$.

By Definition 31, for $z=VC(\mathcal{H})$ we have $\Delta_{\mathcal{H}}(z)=2^{z}.$ So we have $2^{z}\leq z^{m}$ and thus $z\leq m\log z$. This is violated for any $z\geq 3m\log m$, so we must have $VC(\mathcal{H})\leq 3m\log m$.