Learning to represent signals spike by spike

TL;DR

This paper demonstrates that individual neurons can encode precise functional information through learning, challenging the view that only neural populations carry meaningful signals, and shows how networks optimize information transmission with minimal metabolic costs.

Contribution

The study introduces synaptic plasticity rules enabling networks to learn precise spike-based representations, revealing that single-neuron signals are functionally meaningful when optimized for efficiency.

Findings

Single neurons acquire functional meaning through learning.

Networks can represent inputs with maximal accuracy under metabolic constraints.

Single-neuron variability aligns with precise, non-redundant population codes.

Abstract

A key question in neuroscience is at which level functional meaning emerges from biophysical phenomena. In most vertebrate systems, precise functions are assigned at the level of neural populations, while single-neurons are deemed unreliable and redundant. Here we challenge this view and show that many single-neuron quantities, including voltages, firing thresholds, excitation, inhibition, and spikes, acquire precise functional meaning whenever a network learns to transmit information parsimoniously and precisely to the next layer. Based on the hypothesis that neural circuits generate precise population codes under severe constraints on metabolic costs, we derive synaptic plasticity rules that allow a network to represent its time-varying inputs with maximal accuracy. We provide exact solutions to the learnt optimal states, and we predict the properties of an entire network from its input distribution and the cost of activity. Single-neuron variability and tuning curves as typically observed in cortex emerge over the course of learning, but paradoxically coincide with a precise, non-redundant spike-based population code. Our work suggests that neural circuits operate far more accurately than previously thought, and that no spike is fired in vain.

Figures

C, we examine the decoding problem from the point of view of five neurons that seek to represent two input signals. If an input signal lies approximately in the direction of the vector of one of the neurons’ decoding weights, then a few spikes suffice to represent it accurately (Fig. 1Ci). If the input signal lies elsewhere, many more spikes are required to achieve the same accuracy (Fig. 1Cii). For random input signals with zero mean and equal variance, as in Fig. 1Ci, the best representation is achieved when the decoding vectors are evenly distributed. (see Supplementary Information for deta

Bi), and their voltages reflect the smooth, time-varying input (Fig. 2Ei,Fi). Moreover, neurons fire the spikes at roughly the same time from trial to trial. After learning, the membrane potentials are correlated, reflecting their shared inputs, yet the individual spikes are far more susceptible to random fluctuations (Fig. 2Eiii,Fiii). Indeed, whichever neuron happens to fire first immediately inhibits (resets) the others, so that a small initial difference in the membrane potentials is sufficient to change the firing order completely. The random nature of spike timing is therefore a direct c

A). In fact, the respective sign switches simply reflect the immediate sign reversal of the coding error (and thus of the membrane potential) after each new spike. As a result, most connections display some features of the classic STDP rules, e.g. LTP for pre-post pairing, and LTD for post-pre pairing [[39](#bib.bib39), [40](#bib.bib40)]. The only exception are E-E connections that exhibit “reverse STDP”, i.e. potentiation for post-pre pairing (Fig. 3A). Despite their simplicity, these rules are not only spike-time dependent but also weight and voltage-dependent, as observed experimentally [[2

A). As a result, the tuning curves of the learnt network reflect the distribution of inputs experienced by the network (Fig. 4B). In particular, tuning curves are denser and sharper for signal directions that are a-priori more probable. This result is reminiscent of the predictions for efficient rate-based population codes with independent Poisson noise [[20](#bib.bib20)]. Note, however, that our networks learn a spike-per-spike code far more precise and efficient than such rate-based population codes.

A). Despite consisting of 100 neurons that fire at only $\sim 4$Hz the network learns to represent the signals with high precision (Fig. 5B,C). This feat would be impossible if the network had not learnt the strong correlations in speech. As a drawback, the network has become specialized, and a new “non-speech” stimulus results in poor EI balance, high firing rates, and poor coding (Fig. 5D,E). After experiencing the new sound several times, however, the network represents the “non-speech” sound as precisely and parsimoniously as the previously experienced speech sounds (Fig. 5F).

A). Decoding weights are wider and more complex, thus exploiting the high correlations between frequency channels (Fig. 6B). These model predictions are broadly compatible with observations in the mammalian auditory pathway, and notably the representation of speech signals in A1 [[21](#bib.bib21)]. After retraining to the new stimulus, feed-forward weights are modified specifically at the frequencies of the new stimulus (Fig. 6A). However, these changes are not massive. In particular, only a handful of neurons (two in this example) have become truly specialized to the new stimulus, as reflecte

Equations

$$\Delta {\Omega }_{ik}\propto -\beta (V_i+\mu r_i)-{\Omega }_{ik}-\mu {\delta }_{ik}.$$

$$\Delta F_{ij}\propto \alpha x_i-F_{ij},$$

Full text

Learning to represent signals spike by spike

Wieland Brendel,1,2,3,∗ Ralph Bourdoukan,2,∗ Pietro Vertechi,1,2,∗

Christian K. Machens,1,†, Sophie Denève2,†  

1Champalimaud Neuroscience Programme, Champalimaud Foundation, Lisbon, Portugal

2Group for Neural Theory, INSERM U960, Département d’Etudes Cognitives,

Ecole Normale Supérieure, Paris, France

3Werner Reichardt Centre for Integrative Neuroscience,

University of Tübingen, Germany

∗These authors contributed equally

†To whom correspondence should be addressed;

E-mail: sophie.deneve@ens.fr or christian.machens@neuro.fchampalimaud.org

A key question in neuroscience is at which level functional meaning emerges from biophysical phenomena. In most vertebrate systems, precise functions are assigned at the level of neural populations, while single-neurons are deemed unreliable and redundant. Here we challenge this view and show that many single-neuron quantities, including voltages, firing thresholds, excitation, inhibition, and spikes, acquire precise functional meaning whenever a network learns to transmit information parsimoniously and precisely to the next layer. Based on the hypothesis that neural circuits generate precise population codes under severe constraints on metabolic costs, we derive synaptic plasticity rules that allow a network to represent its time-varying inputs with maximal accuracy. We provide exact solutions to the learnt optimal states, and we predict the properties of an entire network from its input distribution and the cost of activity. Single-neuron variability and tuning curves as typically observed in cortex emerge over the course of learning, but paradoxically coincide with a precise, non-redundant spike-based population code. Our work suggests that neural circuits operate far more accurately than previously thought, and that no spike is fired in vain.

Many neural systems encode information by distributing it across the activities of large populations of spiking neurons. A lot of work has provided pivotal insights into the nature of the resulting population codes [1, 2, 3, 4, 5] and their generation through the internal dynamics of neural networks [6, 7, 8, 9]. However, we understand surprisingly little about the precise role of each individual spike in distributing information and in mediating learning.

We revisit this problem by studying a population of excitatory (E) neurons that are interconnected with inhibitory (I) interneurons (Fig. 1. Ai). The excitatory neurons receive many input signals from other neurons within the brain. To encode these signals efficiently, each spike fired by an excitatory neuron should ideally contribute new and unique information to the population code. If each neuron receives a different input signal, this is easy. However, if two excitatory neurons receive similar inputs, they need to communicate with each other so as to not fire spikes for the same type of information. One possibility is that the inhibitory interneurons arbitrate such conflicts by creating competitive interactions between excitatory neurons [10]. How can neurons learn this from experience?

To formalize the problem, we will define a measure for the coding efficiency of a neural population (see Supplementary Information for mathematical details). First, we impose that any downstream area should be able to decode the input signals, $x_{j}(t)$, from a weighted sum of the neural responses, $\hat{x}_{j}(t)=\sum_{k=1}^{N}D_{jk}r_{k}(t)$, where $D_{jk}$ is a decoding weight, and $r_{k}(t)$ is the postsynaptically filtered spike train of the $k$-th excitatory neuron. Second, we assume that the neurons fire as few spikes as possible, or, more generally, that they minimize a cost associated with firing, which we denote by $C(r)$. In other words, we measure the efficiency of the population code through an objective function that trades off accuracy for cost; this objective function is simply the sum of the coding error and the cost, $E=\sum_{j}(x_{j}-\hat{x}_{j})^{2}+C(r)$.

The key problem is that a single excitatory neuron has no access to this global objective function. Rather, it has access to the input signals that arrive via feedforward synapses, $F_{ij}$, and to the filtered spike trains of other neurons that arrive via recurrent synapses, $\Omega_{ik}$. However, imagine that we could set these recurrent synapses to be the feedforward weights multiplied by the decoding weights, so that $\Omega_{ik}=\sum_{j}F_{ij}D_{jk}$. If our neurons are leaky integrate-and-fire neurons, and if we treat the inhibitory interneurons as simple relays for now (Fig. 1. Aii,iii), then the membrane potential of each neuron becomes V_{i}(t)=\sum_{j}F_{ij}\big{(}x_{j}(t)-\hat{x}_{j}(t)\big{)}. Accordingly, the membrane potential now reflects a part of the global coding error, despite being computed from only feedforward and recurrent inputs. Each time this error becomes too large, the membrane potential reaches threshold. The neuron fires, updates the decoded input signal, and thereby decreases the error, as reflected in the voltage reset after a spike. Furthermore, through the recurrent synapses, the neuron will communicate the change in the global coding error to all neurons with similar feedforward inputs. In turn, any excitatory feedforward input into a neuron will immediately be counterbalanced by a recurrent inhibitory input (and vice versa). This latter reasoning links the precision of each neuron’s code to the known condition of excitatory and inhibitory balance (EI balance) [11, 12, 13, 14, 15]. Indeed, balancing excitatory and inhibitory inputs optimally would minimize the variance of the membrane potential, and thus, the error projected in the direction of each neuron’s feedforward weights. In other words, EI balance ensures that V_{i}(t)=\sum_{j}F_{ij}\big{(}x_{j}(t)-\hat{x}_{j}(t)\big{)}\approx 0 [10, 16].

How can a network of neurons learn to move into this very specific regime? Several learning rules for EI balance have been successfully proposed before [17, 18], and spike-timing-dependent plasticity (STDP) can even balance EI currents on a short time scale [18]. However, here we both need to balance EI currents as precisely as possible, and we need to ensure convergence onto the right type of recurrent connectivity (Fig. 1. Aii,iii). We will first examine the problem of EI balance more carefully by studying a neuron’s membrane potential directly after it receives an inhibitory spike from one of its recurrent connections (Fig. 1. Bi; black trace). After the inhibitory spike (Fig. 1. Bi, red), the neuron integrates its (excitatory) feedforward input currents, leading to a transfer of electric charges across the membrane. The arrival of the second (inhibitory) spike (Fig. 1. Bi, blue) then causes a transfer of charges in the opposite direction, which ideally should cancel the total charge accumulated through the feedforward inputs. When the inhibitory spike overshoots (undershoots) its target, then the respective synapse needs to be weakened (strenghtened), see Fig 1. Bii,iii. This scheme keeps the neuron’s voltage (and thereby the coding error) perfectly in check. The regime can be reached by a simple voltage-based learning rule for the recurrent weights of neuron $i$, applied each time a presynaptic neuron $k$ spikes,

[TABLE]

Here $V_{i}$ is the postsynaptic membrane potential before the arrival of the presynaptic spike, while $\beta$ and $\mu$ are positive terms that implement a possible cost factor $C(r)$ (see Supplementary Information).

Fig. 2 illustrates the effect of this learning rule in a network with 20 neurons receiving two random, time-varying inputs. Here the network was initialized with lopsided feed-forward weights and with recurrent weights equal to zero (Fig. 2. Bi). While the network receives the random inputs, the recurrent synapses change according to the learning rule, Eq. 1, and each neuron thereby learns to balance its inputs. Once learnt, the recurrent connectivity reaches the desired structure, $\Omega_{ik}=-\sum_{j}F_{ij}D_{jk}$ for some $D_{jk}$, and the voltages of the neurons become proportional to part of the coding error (see Supplementary Information for convergence proof). As a result of the EI balance, the voltages fluctuations of individual neurons are much better bounded around the resting potential (Fig. 2. Eii), the global coding error decreases (Fig. 2. Aii,Cii), and the network experiences a large drop in the overall firing rates (Fig. 2. Aii,Dii).

Despite these overall improvements, however, the network still fails to represent part of the input, even after the recurrent connections have been learnt (Fig. 2. Cii, arrow). Indeed, in the example provided, this part of the input signal cannot be properly represented because the feedforward connections do not cover the full two-dimensional signal space (Fig. 2. Bii), which becomes particularly evident in the tuning curves of the individual neurons (Fig. 2. Gii).

Consequently, the feedforward connections need to change as well, so that all parts of the input space are properly covered. We can again obtain a crucial insight by considering the final, ‘learnt’ state, in which case the feedforward connections are directly related to the optimal decoding weights. For example, for uncorrelated inputs, the optimal feed-forward and decoding weights are equal, i.e. $F_{ik}=D_{ki}$ (see Supplementary Information). In Fig. 1C, we examine the decoding problem from the point of view of five neurons that seek to represent two input signals. If an input signal lies approximately in the direction of the vector of one of the neurons’ decoding weights, then a few spikes suffice to represent it accurately (Fig. 1. Ci). If the input signal lies elsewhere, many more spikes are required to achieve the same accuracy (Fig. 1. Cii). For random input signals with zero mean and equal variance, as in Fig. 1. Ci, the best representation is achieved when the decoding vectors are evenly distributed. (see Supplementary Information for details and convergence proofs). For correlated inputs, the decoding weights should provide optimal coverage by favoring more frequent input signal directions (See Fig. 1. Cii and Supplementary Information).

The feedforward weights of neuron $j$ can learn to optimally cover the input space if they change each time neuron $j$ fires a spike,

[TABLE]

where $x_{i}$ is the feed-forward input signal, and $\alpha>0$ is a scaling factor. In the case of correlated inputs, the term “$F_{ij}$” is replaced by the co-variance of pre and post-synaptic input currents.

From the perspective of standard frequency-modulated plasticity, the learning rule is Hebbian in that connections are reinforced for co-occurring high pre- and postsynaptic activity. Because of the competition introduced by the recurrent connections, a post-synaptic spike occurs only if no other neuron fired first in response to the same signal. This introduces repulsion between the feedforward weights of different neurons and eventually leads to an even coverage of the input space.

The effect of the feedforward plasticity rule is shown in Fig. 2. Aiii–Giii. The feedforward weights change until the input space is spanned more uniformly (Fig. 2. Biii). While these changes are occuring, the recurrent weights remain plastic and keep the system in a balanced state. At the end of learning, the neuron’s tuning curves are uniformally distributed (Fig. 2. Giii), and the quality of the representation becomes optimal for all input signals (Fig. 2. Aiii,Ciii).

Importantly, the final population code represents the input signals spike by spike, with a precision that approaches the discretization limit imposed by the spikes. Initially, the neurons are unconnected (Fig. 2. Bi), and their voltages reflect the smooth, time-varying input (Fig. 2. Ei,Fi). Moreover, neurons fire the spikes at roughly the same time from trial to trial. After learning, the membrane potentials are correlated, reflecting their shared inputs, yet the individual spikes are far more susceptible to random fluctuations (Fig. 2. Eiii,Fiii). Indeed, whichever neuron happens to fire first immediately inhibits (resets) the others, so that a small initial difference in the membrane potentials is sufficient to change the firing order completely. The random nature of spike timing is therefore a direct consequence of a mechanism that prevents any redundant (or synchronous) firing. More generally, any source of noise or dependency on previous spike history will change the firing order, but without a significant impact on the precision of the code. Thus, variable spike trains co-exist with a highly reproducible and precise population code.

Fortunately, the same results can be obtained in networks with separate excitatory (E) and inhibitory (I) populations (Fig. 1. Ai). In this more realistic case, the inhibitory population must simply learn to represent the population response of the excitatory population, after which it can balance the excitatory population in turn. This can be achieved if we train the EI connections using the feedforward rule (Eq. 2) while the II, EE, and IE connections are trained using the recurrent rule (Eq. 1; see Supplementary Information for details).

Fig. 3 illustrates how the key results obtained in Fig. 2 hold in the the full EI network. The network converges to the optimal balanced state (Fig. 3B), and the precision of the representation improves substantially and approaches the discretization limit (Fig. 3. Bi, Cii), despite the overall decrease in output firing rates (Fig. 3. Bii, Cii). Initially regular and reproducible spike trains (Fig. 3. Biii) become asynchronous, irregular, and comparable to independent Poisson processes (Fig. 3. Biii, pairwise correlations are smaller than 0.001). Furthermore, we observe that the neuron’s tuning curves, when measured along the first two signal directions, are bell-shaped just as in the previous example (Fig. 3. Dii). Note that the inhibitory neurons fire more and have broader tuning than the excitatory neurons. This result holds independent of the chosen initial state of the network, and is simply owed to their smaller number.

Finally, we tested whether the network was robust to other limitations of biological microcircuits, including strong noise injected in the membrane potential, realistic synaptic dynamics with transmission delays, or even randomly removing half of the connections between neurons. We found that under a wide range of conditions, the network learnt to achieve a performance near the discretization limit, outperforming conventional spiking networks or rate-based population coding models (see supplementary Fig. S1). Our findings suggest that, despite being derived for an all-to-all connected network with instantaneous synapses, the learning rules can tolerate large deviations from such initial conditions. This robustness is inherited from the generality of the relationship between E/I balance and efficiency and the subsequent error-correcting coding strategy in the network [19].

We have so far considered uncorrelated inputs. For correlated input signals, the network learns to align its decoding weights to the more frequent signal directions (Fig. 4A). As a result, the tuning curves of the learnt network reflect the distribution of inputs experienced by the network (Fig. 4B). In particular, tuning curves are denser and sharper for signal directions that are a-priori more probable. This result is reminiscent of the predictions for efficient rate-based population codes with independent Poisson noise [20]. Note, however, that our networks learn a spike-per-spike code far more precise and efficient than such rate-based population codes.

To further demonstrate the power of the learning rules, we trained a network to represent speech signals, filtered through 25 frequency channels, in its spiking output (Fig. 5A). Despite consisting of 100 neurons that fire at only $\sim 4$Hz the network learns to represent the signals with high precision (Fig. 5B,C). This feat would be impossible if the network had not learnt the strong correlations in speech. As a drawback, the network has become specialized, and a new “non-speech” stimulus results in poor EI balance, high firing rates, and poor coding (Fig. 5D,E). After experiencing the new sound several times, however, the network represents the “non-speech” sound as precisely and parsimoniously as the previously experienced speech sounds (Fig. 5F).

After training with the speech signals, the feedforward and decoding weights adopt a structure reflecting the natural statistics of speech. The feedforward weights typically have excitatory subfields covering a limited range of frequencies, as well as inhibitory subfields (Fig. 6A). Decoding weights are wider and more complex, thus exploiting the high correlations between frequency channels (Fig. 6B). These model predictions are broadly compatible with observations in the mammalian auditory pathway, and notably the representation of speech signals in A1 [21]. After retraining to the new stimulus, feed-forward weights are modified specifically at the frequencies of the new stimulus (Fig. 6A). However, these changes are not massive. In particular, only a handful of neurons (two in this example) have become truly specialized to the new stimulus, as reflected by their decoding weights ( Fig. 6B).

The accommodation of the network to the new stimulus is largely mediated by plasticity at the recurrent synapses, whereas the feedforward synapses are less essential. Indeed, turning off feedforward plasticity (but not recurrent plasticity) lets the network reach almost the same performance for the new stimulus, whereas turning off recurrent plasticity (but not feedforward plasticity) can even worsen the coding performance instead of improving it (Fig. 6C).

Since learning the new stimulus relies on the recurrent connections re-balancing the feed-forward inputs, EI balance should directly reflect behavioral performance, a prediction compatible with recent observations in the auditory cortex [22]. As a consequence, blocking inhibitory plasticity during perceptual learning should result in a worsening of EI balance and behavioral performance, while blocking excitatory plasticity should have more moderate effects.

In summary, we have shown how populations of excitatory and inhibitory neurons can learn to efficiently represent a signal spike by spike. This type of unsupervised learning, which includes both principal and independent component analysis as special cases [23], has previously been studied largely in rate networks [24, 25, 26, 27]. Implementations that seek to mimic biology by assuming spiking neurons, recurrent network architectures, or local learning rules have been largely limited to heuristic or approximative approaches [28, 29, 30, 31]. Using a rigorous top-down approach, we have here derived biologically plausible rules that are guarantueed to converge to a specific connectivity and achieve a maximally efficient code. Importantly, single spikes are not to be considered as random samples from a rate, but are rather an integral part of a metabolically efficient brain.

While limited here to the representation of time-varying signals, our framework provide a solid starting point to move to other types of computations. Indeed, a precise and efficient neural code is a necessary condition for any precise neural computation. For example, we showed previously that a second set of slower connections can implement arbitrary linear dynamics in designed, optimal networks [10]. The framework presented here can provide crucial intuitions for the learning of these connections as well, since it shows how to represent global errors in local quantities such as voltages.

Apart from these theoretical advances, many of the critical features that are hallmarks of cortical dynamics follow naturally from our framework, even though they were not included in the original objective. We list four of the most important features. First, the predicted spike trains are highly irregular and variable, which has indeed been widely reported in cortical neurons [32, 5]. However, this variability is a signature of the network’s coding efficiency, rather than detrimental [14] or purposeful noise [33, 34]. Second, despite this spike train variability, the membrane potentials of similarly tuned neurons are strongly correlated (due to shared inputs), as has indeed been found in various sensory areas [35, 36]. Third, local and recurrent inhibition in our network serves to balance the excitatory feedforward inputs on a very fast time scale. Such EI balance, in which inhibitory currents track excitatory currents on a millisecond time scale has been found in various systems and under various conditions [37, 38]. Fourth, we have derived learning rules whose polarity depends on the relative timing of pre-and post-synaptic spikes (see insets in Fig. 3A). In fact, the respective sign switches simply reflect the immediate sign reversal of the coding error (and thus of the membrane potential) after each new spike. As a result, most connections display some features of the classic STDP rules, e.g. LTP for pre-post pairing, and LTD for post-pre pairing [39, 40]. The only exception are E-E connections that exhibit “reverse STDP”, i.e. potentiation for post-pre pairing (Fig. 3A). Despite their simplicity, these rules are not only spike-time dependent but also weight and voltage-dependent, as observed experimentally [29].

Our framework thereby bridges from the essential biophysical quantities, such as the membrane voltages of the neurons, to the resulting population code, while providing crucial new insights on learning and coding in spiking neural networks.

References and Notes

• [1]

A. P. Georgopoulos, A. B. Schwartz, R. E. Kettner, Science 233, 1416 (1986).

• [2]

E. P. Simoncelli, B. A. Olshausen, Annual review of neuroscience 24, 1193 (2001).

• [3]

E. Schneidman, M. J. Berry, R. Segev, W. Bialek, Nature 440, 1007 (2006).

• [4]

B. B. Averbeck, P. E. Latham, A. Pouget, Nature Reviews Neuroscience 7, 358 (2006).

• [5]

A. Wohrer, M. D. Humphries, C. K. Machens, Progress in neurobiology 103, 156 (2013).

• [6]

S.-i. Amari, Biological cybernetics 27, 77 (1977).

• [7]

R. Ben-Yishai, R. L. Bar-Or, H. Sompolinsky, Proceedings of the National Academy of Sciences 92, 3844 (1995).

• [8]

C. Eliasmith, Neural computation 17, 1276 (2005).

• [9]

Y. Burak, I. R. Fiete, Proceedings of the National Academy of Sciences 109, 17645 (2012).

• [10]

M. Boerlin, C. K. Machens, S. Denève, Plos Computiational Biology 9, e1003258 (2013).

• [11]

S. Denève, C. K. Machens, Nature neuroscience 19, 375 (2016).

• [12]

C. van Vreeswijk, H. Sompolinsky, Science 274, 1724 (1996).

• [13]

D. J. Amit, N. Brunel, Cerebral cortex 7, 237 (1997).

• [14]

M. N. Shadlen, W. T. Newsome, The Journal of neuroscience 18, 3870 (1998).

• [15]

A. Renart, et al., science 327, 587 (2010).

• [16]

M. Boerlin, S. Denève, PLoS Comput Biol 7, e1001080 (2011).

• [17]

S. Song, K. D. Miller, L. F. Abbott, Nature neuroscience 3, 919 (2000).

• [18]

T. Vogels, H. Sprekeler, F. Zenke, C. Clopath, W. Gerstner, Science 334, 1569 (2011).

• [19]

D. T. Barrett, S. Denève, C. K. Machens, eLife 5, e12454 (2016).

• [20]

D. Ganguli, E. Simoncelli, Neural Computation 26, 2103 (2014).

• [21]

N. Mesgarani, S. David, J. Fritz, S. Shamma, Proc Natl Acad Sci 111, 6792 (2014).

• [22]

B. J. Marlin, M. Mitre, J. A. D’amour, M. V. Chao, R. C. Froemke, Nature 520, 499 (2015).

• [23]

A. Hyvärinen, J. Karhunen, E. Oja, Independent component analysis, vol. 46 (John Wiley & Sons, 2004).

• [24]

E. Oja, Journal of mathematical biology 15, 267 (1982).

• [25]

A. J. Bell, T. J. Sejnowski, Neural computation 7, 1129 (1995).

• [26]

P. Vertechi, W. Brendel, C. K. Machens, Advances in Neural Information Processing Systems (2014), pp. 3653–3661.

• [27]

C. Pehlevan, D. Chklovskii, Advances in Neural Information Processing Systems (2015), pp. 2269–2277.

• [28]

C. Savin, P. Joshi, J. Triesch, PLoS Comput Biol 6, e1000757 (2010).

• [29]

C. Clopath, L. Büsing, E. Vasilaki, W. Gerstner, Nature neuroscience 13, 344 (2010).

• [30]

J. Zylberberg, J. T. Murphy, M. R. DeWeese, PLoS Comput Biol 7, e1002250 (2011).

• [31]

P. D. King, J. Zylberberg, M. R. DeWeese, The Journal of Neuroscience 33, 5475 (2013).

• [32]

D. J. Tolhurst, J. A. Movshon, A. Dean, Vision research 23, 775 (1983).

• [33]

J. Fiser, P. Berkes, G. Orbán, M. Lengyel, Trends in cognitive sciences 14, 119 (2010).

• [34]

L. Buesing, J. Bill, B. Nessler, W. Maass, PLoS Comput Biol 7, e1002211 (2011).

• [35]

J. F. Poulet, C. C. Petersen, Nature 454, 881 (2008).

• [36]

J. Yu, D. Ferster, Neuron 68, 1187 (2010).

• [37]

J. S. Isaacson, M. Scanziani, Neuron 72, 231 (2011).

• [38]

M. Xue, B. V. Atallah, M. Scanziani, Nature 511, 596 (2014).

• [39]

N. Caporale, Y. Dan, Annu. Rev. Neurosci. 31, 25 (2008).

• [40]

D. E. Feldman, Neuron 75, 556 (2012).

• [41]

P. Yin, J. Fritz, S. Shamma, J Neurosci 34, 4396 (2014).

Supplementary figure S1: Robustness of the framework to noise, synaptic delays, and missing connections. Spike trains (black dots) of 60 neurons (out of a total of 200), input signals (thin blue lines), network estimates (thick red lines), and error ratio between the perturbed and optimal networks after learning. (i) White noise current is injected into each neuron. (ii) In addition to the injected noise, synaptic input currents are modeled with a realistic post-synaptic potential, including a transmission delay (inset). (iii) In addition to noise and realistic synapses, 50% of the recurrent connections are randomly removed. (iv) All recurrent connections are removed so that the network is now composed of unconnected leaky integrate and fire neurons. (v) A population of independent Poisson-firing neurons with instantaneous firing rates identical to the network in (i), but without learning.