Deep learning with asymmetric connections and Hebbian updates

TL;DR

This paper demonstrates that deep neural networks can be trained effectively using Hebbian updates with asymmetric feedback connections, achieving performance comparable to back-propagation on challenging image datasets, and explores biologically plausible learning rules.

Contribution

It introduces a Hebbian learning framework with asymmetric, locally updated feedback weights that perform comparably to back-propagation, even with random initializations and untied convolutional layers.

Findings

Hebbian updates achieve similar performance to back-propagation.

Performance remains robust with random, asymmetric feedback weights.

Untied convolutional layers perform comparably to tied weights.

Abstract

We show that deep networks can be trained using Hebbian updates yielding similar performance to ordinary back-propagation on challenging image datasets. To overcome the unrealistic symmetry in connections between layers, implicit in back-propagation, the feedback weights are separate from the feedforward weights. The feedback weights are also updated with a local rule, the same as the feedforward weights - a weight is updated solely based on the product of activity of the units it connects. With fixed feedback weights as proposed in Lillicrap et. al (2016) performance degrades quickly as the depth of the network increases. If the feedforward and feedback weights are initialized with the same values, as proposed in Zipser and Rumelhart (1990), they remain the same throughout training thus precisely implementing back-propagation. We show that even when the weights are initialized differently and at random, and the algorithm is no longer performing back-propagation, performance is comparable on challenging datasets. We also propose a cost function whose derivative can be represented as a local Hebbian update on the last layer. Convolutional layers are updated with tied weights across space, which is not biologically plausible. We show that similar performance is achieved with untied layers, also known as locally connected layers, corresponding to the connectivity implied by the convolutional layers, but where weights are untied and updated separately. In the linear case we show theoretically that the convergence of the error to zero is accelerated by the update of the feedback weights.

Figures

Equations

$$x_{l,i}=\sigma (h_{l,i}),h_{l,i}=$$

$$\text{or}{h}_l=$$

$$\hat{c}(x_0)={\text{argmax}}_i{x}_{L,i}={\text{argmax}}_i\mathcal{N}(x_0,\mathcal{W}).$$

$$W_{l,ij}=W_{l,ij}-\eta \partial \mathcal{L}/\partial W_{l,ij},$$

$$\frac{\partial \mathcal{L}}{\partial W_{l,ij}}=$$

$$\frac{\partial \mathcal{L}}{\partial h_{l,i}}=$$

$$\frac{\partial \mathcal{L}}{\partial W_{l,ij}}=$$

$${\delta }_{l,i}=$$

$$\text{or}{\delta }_l=$$

$${\delta }_{l,i}={\sigma }^{\prime }(h_{l,i})\sum _{k=1}^{n_{l+1}}{\delta }_{l+1,k}{R}_{l+1,ik}.$$

$$\sigma (h)=\max (-1,\min (1,h)),$$

$$|h_{l,i}|\ge 1,\text{or }|x_{l,i}|=1.$$

$$\text{softmax}(x_L{)}_c=p_c=\frac{e^{x_{L,c}}}{{\sum }_{i=1}^C{e}^{x_{L,i}}},c=1,\dots ,C.$$

$$\mathcal{L}(x_L,y)=-\sum _{i=1}^C{x}_{L,i}{y}_i+\log \sum _{i=1}^C{e}^{x_{L,i}},$$

$${\delta }_{L,i}=\frac{\partial \mathcal{L}(x_L,y)}{\partial x_{L,i}}=y_i-p_i.$$

$$\mathcal{L}(x_L,y)=\max (1-x_{L}y,0).$$

$$\frac{\partial \mathcal{L}}{\partial x_L}=\{\begin{array}{ll}-y& \text{if }y\cdot x_L\le 1\\ 0& \text{otherwise}.\end{array}$$

$$\frac{\partial \mathcal{L}}{\partial W_i}=\{\begin{array}{ll}-x_{L-1,i}& \text{mbox}ify=1\text{mbox}and{W}^t{x}_0\le 1\\ x_{L-1,i}& \text{mbox}ify=-1\text{mbox}and{W}^t{x}_{L-1}\ge -1\\ 0& \text{mbox}otherwise\end{array},$$

$$\mathcal{L}(x_L,y)=\max (1-x_{L,c},0)+\mu \sum _{i\ne c}\max (1+x_{L,i},0).$$

$$\frac{\partial \mathcal{L}(x_L,y)}{\partial x_{L,i}}=\{\begin{array}{ll}-1& \text{ if }i=c\text{ and }{x}_{L,i}\le 1\\ \mu & \text{ if }i\ne c\text{ and }{x}_{L,i}\ge -1\\ 0& \text{ otherwise.}\end{array}$$

$${\delta }_{L,i}=\{\begin{array}{ll}1& \text{ if }i=c\text{ and }{x}_{L,i}\le 1\\ -\mu & \text{ if }i\ne c\text{ and }{x}_{L,i}\ge -1\\ 0& \text{ otherwise.}\end{array}$$

$$C(W_1,W_2)=|Y-W_2{W}_{1}X{|}^2.$$

$${\dot{W}}_2=$$

$${\dot{W}}_1=$$

$${\dot{W}}_2=$$

$${\dot{W}}_1=$$

$${\dot{R}}_2=$$

$${\dot{W}}_2=$$

$${\dot{W}}_1=$$

$${\dot{W}}_k$$

$$⋮$$

$${\dot{W}}_i$$

$$⋮$$

$${\dot{W}}_1$$

$$H_c=h_c+2M{s}_c-2M{t}_c,$$

$${\sigma }_{\delta }(x)=\{\begin{array}{ll}1+ϵ& \text{if }x>2M+S\\ 1& \text{if }x\in [M,2M+S]\\ 0& \text{if }x\in [-M,M]\\ -\mu & \text{if }x\in [-2M-S,-M]\\ -\mu -ϵ& \text{if }x<-2M-S\end{array},{\sigma }_t(x)=\{\begin{array}{ll}1& \text{if }x>1\\ 0& \text{if }-\mu \le x\le 1\\ -1& \text{if }x<-\mu \end{array}$$

Full text

1 Introduction

The success of multi-layer neural networks (deep networks) in a range of prediction tasks as well some observed similarities observed between the properties of the network units and cortical units ([29]), has raised the question of whether they can serve as models for processing in the cortex [10, 18]. The feedforward architecture of these networks is clearly consistent with models of neural computation: a hierarchy of layers, where the units in each layer compute their activity in terms of the weighted sum of the units of the previous layer. The main challenge with respect to biological plausibility is in the way these networks are trained.

Training of feedforward networks is based on a loss function that compares the output of the top layer of the network to a target. Small random subsets of training data are then used to compute the gradient of the loss with respect to the weights of the network, and these are then updated by moving a small distance in the opposite direction of the gradient. Due to the particular structure of the function represented by these multi-layer networks the gradient is computed using back-propagation - an algorithmic formulation of the chain rule for differentiation [21]. In the feedforward step the input is passed bottom-up through the layers of the network to produce the output of the top layer and the loss is computed. Back-propagation proceeds top-down through the network. Successively two things occur in each layer: first, the unit activity in the layer is updated in terms of the layer above - feedback, then the weights feeding into this layer are updated. The gradient of each weight is a product of the activity of the units it connects - the feedforward pre-synaptic activity of the input unit in the lower layer and the feedback activity in the post-synaptic unit in the current layer. In that sense the gradient computation has the form of local Hebbian learning. However, a fundamental element of back-propagation is not biologically plausible as explained in [30, 17]. The feedback activity of a unit is computed as a function of the units in the layer above it in the hierarchy in terms of the same weight matrix used to compute the feedforward signal, implying a symmetric synaptic connectivity matrix.

Symmetry of weight connection is an unrealistic assumption. Although reciprocal physical connections between neurons are more common than would be expected at random, these connections are physically separated in entirely different regions of the neuron and can in no way be the same. The solution proposed both in [30] and in [17] is to create a separate system of feedback connections. The latter model is simpler in that the feedback connections are not updated so that the top-down feedback is always computed with the same weights. The earlier model proposes to update the feedback weights with the same increment as the feedforward weights, which as mentioned above has a Hebbian form. Assuming they are initialized with the same values, they will always have the same value. This guarantees that the back-propagation computation is executed by the network, but in effect reintroduces exact weight symmetry in the back-door, and is unrealistic. In contrast, the computation in [17] does not replicate back-propagation, as the feedback weights never change, but the price paid is that in deeper networks it performs quite poorly.

The main contribution of this paper is to experiment with the idea proposed in [30], but initialize the feedforward and feedback weights randomly (thus differently). We call this updated random feedback (URFB). We show that even though the feedback weights are never replicates of the feedforward weights, the network performance is comparable to back-propagation, even with deep networks on challenging benchmark dataset such as CIFAR10 and CIFAR100 [11]. In contrast, the performance with fixed weights -fixed random feedback (FRFB), as in [17], degrades with depth. It was noted in [17] that in shallow networks the feedforward weights gradually align with the fixed feedback weights so that in the long run an approximate back-propagation is being computed, hence the name feedback alignment. We show in a number of experiments that this alignment phenomenon is much stronger in URFB even for deep networks. However we also show that from the very initial iterations of the algorithm, long before the weights have aligned, the evolution of both the training and validation errors is comparable to that of back-propagation.

In our experiments we replace the commonly used unbounded rectified linear unit, with a saturated linearity $\sigma(x)=\min(\max(x,-1),1)$, which is more biologically plausible, as it is not unbounded, we avoid normalization layers whose gradient is quite complex and not easily amenable to neural computation, and we run all experiments with the simplest stochastic gradient descent that does not require any memory of earlier gradients. We also experiment with randomly zeroing out half of the connections, separately for feedforward and feedback connections. Thus not only are the feedforward and feedback weights different, but connectivity is asymmetric. In a simplified setting we provide a mathematical argument for why the error decreases faster with updated feedback weights compared to fixed feedback weights.

Another issue arising in considering the biological plausibility of multilayer networks is how the teaching signal is incorporated in learning. The primary loss used for classification problems in the neural network literature is the cross-entropy of the target with respect to the softmax of the output layer (see section 3.2). The first step in back-propagation is computing the derivative of this loss with respect to the activities of the top layer. This derivative, which constitutes the feedback signal to the top layer, involves the computation of the softmax - a ratio of sums of exponentials of the activities of all the output units. It is not a local computation and is difficult to model with a neural network. As a second contribution we experiment with an alternative loss, motivated by the original perceptron loss, where the feedback signal is computed locally only in terms of the activity of the top-level unit and the correct target signal. It is based on the one-versus all method commonly used with support vector machines in the mult-class setting and has been implemented through network models in [1, 13, 2].

Finally, although convolutional layers are consistent with the structure of retinotopic layers in visual cortex, back-propagation through these layers is not biologically plausible. Since the weights of the filters applied across space are assumed identical, the gradient of the unique filter is computed as the sum of the gradients at each location. In the brain the connections corresponding to different spatial locations are physically different and one can’t expect them to undergo coordinated updates, see [3]. This leads us to the final set of experiments where instead of purely convolutional layers we use a connectivity matrix that has the sparsity structure inherited from the convolution but the values in the matrix are ‘untied’ undergo independent local updates. Such layers are also called locally connected layers and have been used in [3] in experiments with biologically plausible architectures. The memory requirements of such layers are much greater than for convolutional layers, as is the computation, so for these experiments we restrict to simpler architectures. Overall we observe the same phenomena as with convolutional layers, namely the update of the feedback connections yields performance close to that of regular back-propagation.

The paper is organized as follows. In section 2 we describe related work. In section 3 we describe the structure of a feedforward network, the back-propagation training algorithm and explain how it is modified with separate feedback weights. We describe the loss function and explain why it requires only local Hebbian type updates. In section 4 we report a number of experiments and illustrate some interesting properties of these networks. We show that performance of URFB is lower but close to back-propagation even in very deep networks, on more challenging data sets that actually require a deep network to achieve good results. We show that using locally-connected layers works, although not as well as convolutional networks, and that the resulting filters although not tied apriori show significant similarity across space. We illustrate the phenomenon of weight alignment that is much more pronounced in URFB. In section 5 we describe a simplified mathematical framework to study the properties of these algorithms and show some simulation results that verify that updating the feedback connections yields faster convergence than fixed feedback connections. We conclude with a discussion. Mathematical results and proofs are provided in the Appendix.

2 Related work

As indicated in the introduction, the issue of the weight symmetry required for feedback computation in back-propagation, was already raised by [30] and the idea of separating the feedback connections from the feedforward connections was proposed. They then suggested updating each feedforward connection and feedback connection with the same increment. Assuming all weights are initialized at the same value the resulting computation is equivalent to back-propagation. The problem is that this reintroduces the implausible symmetry since the feedback and feedforward weights end up being identical.

In [17] the simple idea of having fixed random feedback connections was explored and found to work well for shallow networks. However, the performance degrades as the depth of the network increases. It was noted that in shallow networks the feedforward weights gradually align with the fixed feedback weights so that in the long run an approximate back-propagation is being computed, hence the name feedback alignment. In [16] the performance degradation of feedback alignment with depth was addressed by using layer-wise normalization of the outputs. This yielded results with fixed random feedback FRFB that are close to momentum based gradient descent of the back-propagation algorithm for certain network architectures. However the propagation of the gradient through the normalization layer is complex and it is unclear how to implement it in a network. Furthermore [16] showed that a simple transfer of information on the sign of the actual back-propagation gradient yields an improvement on using the purely random back-propagation matrix. It is however unclear how such information could be transmitted between different synapses.

In [28] a model for training a multilayer network is proposed using a predictive coding framework. However it appears that the model assumes symmetric connections, i.e. the strength of the connection from an error node and a variable in the preceding layer is the same as the reverse connection. A similar issue arises in [20], where in the analysis of their algorithm, they assume that in the long run, since the updates are the same, the synaptic values are the same. This is approximately true, in the sense that the correlations between feedforward and feedback weights increase but significant improvement in error rates are observed even early on when the correlations are weak.

[4] implements a proposal similar to [30] in the context of an autoencoder and attempts to find STDP rules that can implement the same increment for the feedforeward and feedback connections. Again it is assumed that the initial conditions are very similar so that at each step the feedforward and feedback weights are closely aligned.

In a recently archived paper [19] also goes back to the proposal in [30]. However, as in our paper, they experiment with different initializations of the feedforward and feedback connections. They introduce a pairing of feedback and feedforward units to model the gating of information from the feedforward pass and the feedback pass. Algorithmically, the only substantial difference to our proposal is in the error signal produced by the output layer, only connections to the output unit representing the correct class are updated.

Here we show that there is a natural way to update all units in the output layer so that subsequent synaptic modifications in the back-propagation are all Hebbian. The correct class unit is activated at the value 1 if the input is below a threshold, and the other classes are activated as $-\mu$ if the input is above a threshold. Thus, corrections occur through top-down feedback in the system when the inputs of any of the output units are not of sufficient magnitude and of the correct sign. We show that this approach works well even in much deeper networks with several convolutional layers and with more challenging data sets. We also present a mathematical analysis of the linearized version of this algorithm and show that the error converges faster when the feedback weights are updated compared to when they are held fixed as in [17].

[15, 3] study target propagation where an error signal is computed in each hidden unit as the difference between the feedforward activity of that unit and a target value propagated from above with feedback connections that are separate from the feedforward connections. The feedback connections between each two consecutive layers are trained to approximate the inverse of the feedforward function between those layers, i.e. the non-linearity applied to the linear transformation of the lower layer. In [3] they analyze the performance of this method on a number of image classification problems and use locally connected layers instead of convolutional layers. In target propagation the losses for both the forward and the backward connections rely on magnitudes of differences between signals requiring a more complex synaptic modification mechanism than simple products of activities of pre and post synaptic neurons as proposed in our model.

Such synaptic modification mechanisms are studied in [7]. A biological model for the neuronal units is presented that combines the feedforward and feedback signals within each neuron, and produces of an error signal assuming fixed feedback weights as in [17]. The idea is to divide the neuron into two separate compartments one computing feedforward signals and one computing feedback signals, with different phases of learning involving different combinations of these two signals. In addition to computing an error signal internally to the neuron this model avoids the need to compute signed errors, which imply negative as well as positive neuronal activity. However this is done by assuming the neuron can internally compute the difference in average voltage between two time intervals. In [22] this model is extended to include an inhibitory neuron attached to each hidden unit neuron with plastic synaptic connections to and from the hidden unit. They claim that this eliminates the need to compute the feedback error in separate phases form the feedforward error.

In our model we simply assume that once the feedforward phase is complete the feedback signal replaces the feedforward signal at a unit - at the proper timing - to allow for the proper update of the incoming feedforward and outgoing feedback synapses.

3 The updated random feedback algorithm

In this section we first describe the structure of a multilayer network, how the back-propagation algorithm works and how we modify it to avoid symmetric connections and maintain simple Hebbian updates to both feedforward and feedback connections. We then describe a loss function, whose derivatives can be computed locally, yielding a Hebbian input dependent update of the weights connecting to the final output layer.

3.1 Updated asymmetric feedback connections

A multi-layer network is composed of a sequence of layers $0,\ldots,L$. The data at the input layer is denoted $x_{0}$. Each layer is composed of $n_{l}$ units. Let $W_{l,ij}$ be the feedforward weight connecting unit $j$ in layer $l-1$ to unit $i$ in layer $l$. Let $x_{l},l=1,\ldots,L$ be the output of layer $l$, this is computed as

[TABLE]

where $\sigma$ is some form of non-linearity and $W_{l}$ is the $n_{l}\times n_{l-1}$ matrix of weights connecting layer $l-1$ to layer $l$. We denote $h_{l,i}$ the input of unit $i$ of layer $l$. For classification problems with $C$ classes the top layer $L$, also called the output layer, has $C$ units $x_{L,1},\ldots,x_{L,C}$. In this last layer no non-linearity is applied, i.e. $x_{L,i}=h_{L,i}.$ For given input $x_{0}$ we can write $x_{L}={\mathcal{N}}(x_{0},{\mathcal{W}})$, where ${\mathcal{N}}$ represents the function computed through the multiple layers of the network with the set of weights ${\mathcal{W}}$. The classifier is then defined as:

[TABLE]

A feedforward network with 3 layers is shown in figure 1

We define a loss ${\mathcal{L}}(x_{L},y,{\mathcal{W}})$ comparing the activity of the output layer to a target value, an indicator vector denoting the correct class of the input. At each presentation of a training example the derivative $\partial{\mathcal{L}}/\partial W_{l,ij}$ of the loss with respect to each weight is computed, and the value of the weight is updated as

[TABLE]

where $\eta$ is a small scalar called the time-step or learning rate. This is done in two phases. In the first phase, the feedforward phase, the input $x_{0}$ is presented at layer $l=0$ and passed successively through the layers $l=1,\ldots,L$ as described in (3.1). In the second phase the derivatives are computed starting with $W_{L,ij}$ for the top layer and successively moving down the hierarchy. At each layer the following two equalities hold due to the chain rule for differentiation:

[TABLE]

If we denote $\delta_{l,i}=\frac{\partial{\mathcal{L}}}{\partial h_{l,i}}$ we can write this as:

[TABLE]

where $\sigma^{\prime}(h_{l})$ is the diagonal matrix with entries $\sigma^{\prime}(h_{l,i})$ on the diagonal. So we see that the update to the synaptic weight $W_{l,ij}$ is the product of the feedback activity at unit $i$ of layer $l$ denoted by $\delta_{l,i}$, also called the error signal, and the input activity from unit $j$ of layer $l-1$. And the feedback activity (error signal) of layer $l$ is computed in terms of the feedforward weights connecting unit $i$ in layer $l$ to all the units in layer $l+1$. This is the symmetry problem.

We now separate the feedforward weights from the feedback weights. Let $R_{l+1,ik}$ be the feedback weight connecting unit $k$ of layer $l+1$ to unit $i$ of layer $l$. The the second equation in (3.1) becomes:

[TABLE]

If $R=W^{t}$ we get the original back-propagation update. We illustrate the general updating scheme computation in figure 2.

In [17] the values of $R$ are held fixed at some random initial value, which we denote fixed random feedback (FRFB). In contrast, in our proposal, since $R_{l+1,ik}$ connects the same units as $W_{l+1,ki}$ it experiences the same pre and post-synaptic activity and so will be updated by the same Hebbian increment - $\delta_{l+1,k}x_{l,i}.$ We call this method updated random feedback - URFB. If the initial values of $R_{l,ik}$ are the same as $W_{l,ki}$ then equality will hold throughout the update iterations resulting in a symmetric system performing precise back-propagation. This is the proposal in [30]. We experiment with different initializations, so that the updates are not performing back-propagation, even in the long run after many iterations the weights are not equal, although their correlation increases. We show that classification rates remain very close to those obtained by back-propagation. In addition, in order to increase the plausibility of the model we also experiment with sparsifying the feedforward and feedback connections by randomly fixing half of each set of weights at 0.

Remark 1: It is important to note that the feedback activity $\delta_{l,i}$ replaces the feedforward activity $x_{l,i}$ and needs to be computed before the update of the feedforward weights feeding into unit $i$ and the feedback weights feeding out of that unit, but using the original feedforward activity $x_{l-1,i}$ of the units in layer $l-1$. This requires a very rigid sequencing of the algorithm from top to bottom.

Remark 2: The feedback signal propagates by computing a linear combination of the feedback signals in the higher layers, but is then multiplied by the term $\sigma^{\prime}(h_{l,i})$. To simplify as much as possible we have employed a non-linearity $\sigma$ of the form

[TABLE]

which is simply a saturated linear function at thresholds $-1$ and $1$, and $\sigma^{\prime}(h)=1$ if $|h|\leq 1$ and 0 otherwise. Thus the feedback activity $\delta_{l,i}$ is the linear combination of the feedback activities $\delta_{l+1,k}$ in the layer above unless

[TABLE]

i.e. bottom-up input $h_{l,i}$ is too high or too low, in which case $\delta_{l,i}=0$. A local network to compute this thresholding is described in Appendix 1. The computation of the top level derivative $\delta_{L,i}=\partial{\mathcal{L}}/\partial h_{L,i}$ will be discussed in the next section.

3.2 Loss function

The softmax loss commonly used in deep learning defines the probability of each output class as a function of the activities $x_{L,i}$ as follows:

[TABLE]

The loss computes the negative log-likelihood of these probabilities:

[TABLE]

where $y_{c}=1$ if the class of the input is $c$ and $y_{i}=0,i\neq c$. Thus the initial feedback signal is:

[TABLE]

This requires the computation of the softmax function $p_{i}$, which involves the activity of all other units, as well as exponentiations and ratios.

The classification loss function used here is motivated by the hinge loss used in standard linear SVMs. In the simplest case of a two class problem we code the two classes as a scalar $y=\pm 1$ and use only one output unit $x_{L}$. Classification is based on the sign of $x_{L}$. The loss is given by

[TABLE]

The derivative of this loss with respect to $x_{L}$, is simply

[TABLE]

The idea is to that the output $x_{L}$ should have the same sign as $y$ and be sufficiently large in magnitude. Once the output achieves that, there is no need to change it and the loss is zero.

Writing $x_{L}=W^{t}x_{L-1}$, this yields the perceptron learning rule with margin (see [24]):

[TABLE]

If we think of the supervised signal as activating the output unit with $\delta_{L}=+1$ for one class and $\delta_{L}=-1$ for the other, unless the input is already of the correct sign and of magnitude greater than 1, then $\delta_{L}=-\partial{\mathcal{L}}/\partial x_{L}$. The update rule can be rewritten as $W_{i}\leftarrow W_{i}+\eta\Delta W_{i}$ where $\Delta W_{i}=\delta_{L}\cdot x_{L-1,i}$ if $x_{L}=W^{t}x_{L-1}$ satisfies $\delta_{L}x_{L}\leq 1.$ In other words if the output $x_{L}$ has the correct sign by more than the margin of 1 then no update occurs, otherwise the weight is updated by the product of the target unit activity and the input unit activity. In that sense the update rule is Hebbian, except for shut down of the update when $x_{L}$ is ‘sufficiently correct’.

One might ask why not use the unconstrained Hebbian update $\Delta W_{i}=\eta\delta_{L}x_{L-1,i}$, which corresponds to a loss that computes the inner product of $y$ and $x$. Unconstrained maximization of the inner product can yield over fitting in the presence of particularly large values of some of the coordinates of $x$ and create an imbalance between the two classes if their input feature distribution is very different. This becomes all the more important with multiple classes, which we discuss next.

For multiple classes we generalize hinge loss as follows. Assume as before $C$ output units $x_{L,1},\ldots,x_{L,C}$. For an example $x$, of class $c$ define the loss

[TABLE]

where $\mu$ is some balancing factor. The derivative has the form:

[TABLE]

Henceforth we will set $\delta_{L,i}=-\partial{{\mathcal{L}}(x_{L},y)}/{\partial x_{L,i}}.$ Substituting the feedback signal $\delta_{L,i}$ for the feedforward signal $x_{L,i}$ at the top layer has the following simple form:

[TABLE]

and equation (3.1) is then applied to compute the feedback to layer $L-1$ - $\delta_{L-1}$ and the update of the weights $W_{L},R_{L}.$ All experiments below use this rule.

Note that $\delta_{L,i}$ is precisely the target signal, except when the feedforward signal has the correct value - greater than 1 if $i=c$ (the correct class) and less than $-\mu$ for $i\neq c$ (the wrong class). This error signal only depends on the target value and input to unit $i$, no information is needed regarding the activity of other units. One can ask whether a neuron can produce such an output, which depends both on the exterior teaching signal and on the feedforward activity. In Appendix 1 we propose a local network that can perform this computation.

This loss produces the well known one-versus-all method for multi-class SVM’s (see for example [8]), where for each class $c$ a two class SVM is trained for class $c$ against all the rest lumped together as one class. Classification is based on the maximum output of the $C$ classifiers. Each unit $x_{L,c}$ can be viewed as a classifier of class $c$ against all the rest. When an example of class $c$ is presented it updates the weights to obtain a more positive output, when an example of any class other than $c$ is presented it updates the weights to obtain a more negative output. Other global multiclass losses for SVM’s can be found in [8]. In [1, 2] a network of binary neurons with discrete synapses was described that implements this learning rule to update connections between discrete neurons in the input and output layers and with positive synapses. Each class was represented by multiple neurons in the output layer. Thus classification was achieved through recurrent dynamics in the output layer, where the class with most activated units maintained sustained activity, whereas activity in the units corresponding to other classes died out.

4 Experiments

We report a number of experiments comparing the updated (URFB) to the fixed feedback matrix (FRFB) and comparing the multi-class hinge loss function to the cross-entropy with softmax loss. We restrict ourselves to image data. Since it is quite easy to obtain good results with the widely used MNIST handwritten data set [14] we focus on two more challenging data sets called CIFAR10 and CIFAR100 [11]. Each dataset contains 32x32 color images from 10 classes for the first and 100 classes for the second. The classes are broadly defined so that the category bird will contain multiple bird types at many different angles and scales. Some sample images are show in figure 3. Each data set has 50000 training images and 10000 test images. An even more challenging data set is IMAGENET http://www.image-net.org/, which contains thousands of classes. We do not experiment with this dataset as the images are much larger and the training set is much larger so that training is time consuming.

There are a number of benchmark network architectures that have been developed over the past decade with good results on these datasets, see ([25, 12, 9]). These networks are very deep and employ a variety of methods to accelerate convergence, such as adaptive time-steps and batch normalization. These improvements involve steps that are not easily modeled as neural computations. For that reason we restrict our learning method to the simplest form of gradient descent with a fixed time step and no normalization. We do not perform any pre-processing of the input data, nor do we employ any methods of data augmentation to improve classification results. All our weights are initialized based on the method described in [6]. Weights are uniformly drawn between $[-b_{l},b_{l}]$ where $b_{l}$ is a function of the number of incoming and outgoing connections to a unit in layer $l$.

In the experiments we demonstrate the following:

With regular back-propagation (BP) hinge loss performs slightly worse than the softmax loss but results are comparable. 2. 2.

For shallow networks URFB performs somewhat better then FRFB but mainly converges faster. It never performs as well as BP but is close. 3. 3.

For deeper networks URFB again performs close to BP but FRFB performance degrades significantly. 4. 4.

With locally connected - untied - layers instead of convolutional layers results are slightly worse overall but the relationship between the different methods is maintained. 5. 5.

In URFB the feedback weights are never the same as the feedforward weights, although the correlation between the two sets of weights increases with iteration. 6. 6.

Even in initial iterations, when the weights are far from being aligned, training and validation error rates decrease at similar rates to back propagation.

We first experiment with a shallow network with only two hidden layers, one convolutional and one fully connected.

simpnet: Conv 32 5x5; Maxpool 3; Drop .8; Full 500; Drop .3; Output

The notation Conv 32 5x5 means that we have 32 - 5x5 filters, each applied as a convolution to the input images, producing 32 output arrays of dimension 32x32. Maxpool 3 means that at each pixel the maximal value in the 3x3 window centered at that pixel is substituted for the original value (padding with 0’s outside the grid), in each of the 32 output arrays, and then only every second pixel is recorded producing 32 arrays of size 16x16. Drop .8 means that for each training batch, a random subset of 80% of the pixels in each array are set to 0 so that no update occurs to the outgoing synaptic weights. This step was introduced in [26] as a way to avoid overfitting to the training set. It is also attractive as a model for biological learning as clearly not all synapses will update at each iteration. Full 500 means a layer with 500 units, each connected to every unit in the previous layer.

The Output layer has $C$ output units corresponding to each class. We use the saturated linearity $\sigma(x)=\min(\max(x,-1),1)$ and the hinge loss function as given in (4). The update is a simple SGD with a fixed time step of .1, and the network is trained for 1000 epochs with batch-size of 500. We make a point to avoid any adaptive normalization layers as these require a complex gradient that is not amenable to simple neural computations. We avoid the more sophisticated time step adaptations which depend on previous updates and some normalizations, which again do not seem amenable to simple neural computations.

The three parameters we adjusted were the time step and two drop out rates. We experimented with time-steps .01, .1 and 1.0 for the simpnet and found the best behavior on a held out validation set of 5000 samples was with the value .1. We kept this value for all further experiments. We had two dropout layers in each network. One between convolutional layers and one before the output layer. The values were adjusted by running a few tens of iterations and making sure the validation loss was closely tracking the training loss.

We also experiment with pruning the forward and backward connections randomly by 50%. In other words half of these connections are randomly set to 0. Error rates for CIFAR10 and CIFAR100 datasets are shown in figure 5. We note that the use of the multi-class hinge loss leads to only a small loss in accuracy relative to softmax. All experiments with random feedback are performed with the hinge loss. For CIFAR10 the difference between R fixed - FRFB - and R updated - URFB - is small, but becomes more significant when connectivity is reduced to 50% and with the CIFAR100 database.

Note that in the simple network the only layer propagating back an error signal is the fully connected layer. The first layer, which is convolutional, does not need to back-propagate an error. The evolution of error rates for the different protocols as a function of iteration can be seen in figure 4.

We experiment with a deep network with multiple convolutional layers, and observe an even larger difference between R fixed and R updated. With the deep network FRFB performs very poorly. The deep architecture is given here.

deepnet: Conv 32 5x5; Maxpool 3; Conv 32 3x3; Conv 32 3x3; Maxpool 3; Drop .8; Conv 32 3x3; Conv 32 3x3; Maxpool 3; Drop .3; Full 500; Output

Finally we try an even deeper network with residual layers as in [9]. This means that after every pair of consecutive convolutional layers at the same resolution we introduce a layer that adds the two previous layers with no trainable parameters. This architecture was found to yield improved results on a variety of datasets.

deepernet: conv 16 3x3; conv 16 3x3; SUM; conv 32 3x3; conv 32 3x3; SUM; maxpool 3; drop .5; conv 64 3x3; conv 64 3x3; SUM; maxpool 3; conv 128 3x3; conv 128 3x3; SUM; maxpool 3; drop .8; full conn. 500; output

We see in figure 6 that for the default BP with softmax or hinge loss the error rate decreases from 50% with deepnet to 42% with deepernet. URFB also shows a decrease in error between deepnet and deepernet and again FRFB performs very poorly. The evolution of error rates for the different protocols as a function of iteration can be seen in figure 7.

4.1 Untying the convolutional layers - locally connected layers

We explore ‘untied’ local connectivities determined by the corresponding convolutional layer. These are also called locally connected layers [3]. A convolution corresponds to multiplication by a sparse matrix where the entry values are repeated in each row, but with some displacement. This again is not plausible because it assumes identical weights across a retinotopic layer. Furthermore the back-propagation update of a particular weight in a convolutional layer computes the sum of all products $\sum_{i}\delta_{l,i}x_{l-1,i+k}$, where $i$ represents locations on the grid and $k$ is a fixed displacement. So, it assumes that each one of the identical weights is updated by information summed across the entire layer.

To avoid these issues with biological plausibility we instead assume each of the entries of the sparse matrix is updated separately with the corresponding product $\delta_{l,i}x_{l-1,i+k}$. Only non-zero elements of the sparse matrix, that correspond to connections implied by the convolutional operation are updated. This is implemented using tensorflow sparse tensor operations, and is significantly slower and requires more memory than the ordinary convolutional layers. The error rates are similar to those with the original convolutional layers even with the deeper networks. In figure 9 for CIFAR10, we show a comparison of error rates between networks with convolutional layers to networks with corresponding untied layers for the different training protocols. We show comparisons for simpnet and deepnet_s defined below.

Despite the fact that the weights are updated without being tied across space, the final connectivity matrix retains a strong spatial homogeneity. In other words at each location of the output layer one can restructure the weights to a filter and inspect how similar these filters are across locations. We presume that this is due to the fact that in the data local structures are consistent across space. In figure 8 we show a couple of these 5x5 filters across four different locations in the 32x32 grid in the trained simpnet. We see that even after 1000 iterations there is significant similarity in the structure of the filters despite the fact that they were updated independently for each location.

We also experiment with a deeper network:

deepnet_s: conv 16 3x3; conv 16 3x3; ΨSUM;maxpool 3, stride 3; drop .5; conv 64 3x3; conv 64 3x3; SUM; maxpool 2, stride 2; conv 64 2x2; conv 64 2x2; SUM; maxpool 2, stride 2; drop .5; full conn. 500; output

Here we could not run all convolutional layers as untied layers due to memory constraints on our GPUs. Instead we ran the network for 100 epochs with the regular convolutional layers, then we froze the first layer and retrained the remaining layers from scratch using the untied architecture, see figure 9. This would mimic a situation where the first convolutional layer perhaps corresponding to V1 has connections that are predetermined and not subject to synaptic modifications. Once more, we see that the untied layers with URFB reach error rates similar to those of the regular convolutional layers with standard gradient descent. And again, we observe that with a deeper network FRFB performance is much worse.

4.2 Weight alignment

One of the claims in [17] is that the network gradually aligns the updated feedforward weights to the fixed feedback weights. In figure 10 we show the evolution of the correlations between the feedforward weights $W_{l}$ and $R_{l}$ for simpnet. Recall that the layer with highest index is the output layer and typically reaches high correlations in both URFB and FRFB. We see, however, that the alignment is much stronger for the URFB. Note that when weights are highly correlated the network is effectively implementing back-propagation.

In figure 11 we again show the evolution of the correlations between $W_{l},R_{l}$ for the seven updated layers of the deeper network deepnet_s. Note that for some but not all layers the final correlations are very close to one. However the training loss and error rates change very rapidly in the initial iterations when the correlations are very low. Interestingly the correlation levels are not a monotone function of layer depth.

5 Mathematical analysis of updated random feedback

The mathematical analysis closely follows the methods developed in [23] and thus focuses on linear networks, i.e. $\sigma(x)=x$ and a simple quadratic loss. We start with a simple two layer network.

Let the input $x\in\mathbb{R}^{n_{0}}$, and the output $y=W_{2}W_{1}x\in\mathbb{R}^{n_{2}}$ with weights $W_{1}\in\mathbb{R}^{n_{1}\times n_{0}},W_{2}\in\mathbb{R}^{n_{2}\times n_{1}}$. If $X$ is the $n_{0}\times N$ matrix of input data and $Y$ the $n_{2}\times N$ of output data the goal is to minimize

[TABLE]

We write $T=YX^{t}\in\mathbb{R}^{n_{2}\times n_{0}}$, and assume that $XX^{t}=I$, namely the input coordinates are uncorrelated. The gradient of $L$ with respect to $W_{1}$ and $W_{2}$ yield the following gradient descent ODE’s, which corresponds to regular back-propagation:

[TABLE]

with some initial condition $W_{1}(0),W_{2}(0).$ If we implement the FRFB or URFB described above we get the following three equations:

[TABLE]

where $R_{2}\in\mathbb{R}^{n_{1}\times n_{2}}$ and $\epsilon$ is a parameter. Setting $\epsilon=0$ corresponds to FRFB, as there is no modification of the matrix $R$. The URFB corresponds to $\epsilon=1.$ Our goal is to show that the larger $\epsilon$ the faster the convergence of the error to 0.

To simplify the analysis of (5) we assume $W_{1}(0)=W_{2}(0)=0$ and $R_{2}(0)$ is random. Then $R_{2}=R_{2}(0)+\epsilon W_{2}^{t}$ and the system reduces to

[TABLE]

For deeper networks, and again assuming the $W_{l}$ matrices are initialized at 0, we have the following equations for URFB:

[TABLE]

where $E=T-W_{k}\cdots W_{1},\quad T\in\mathbb{R}^{n_{k}\times n_{0}}$ and $W_{i}\in\mathbb{R}^{n_{i}\times n_{i-1}},i=1,\ldots,k$. Again our goal is to show that as $\epsilon$ increases from 0 to 1, the error given by $e=tr(E^{t}E)$ converges faster to 0.

The precise statements of the results and the proofs can be found in Appendix 2. Here we show through a simulation that convergence is indeed faster as $\epsilon$ increases from $\epsilon=0$ (FRFB) to $\epsilon=1$ (URFB).

5.1 Simulation

We simulated the following setting. An input layer of dimension 40, two intermediate layers of dimension 100 and an output layer of dimension 10. We assume $X=I_{40}$ so that $T=W^{*}_{1}W^{*}_{2}W^{*}_{3}$ with $W^{*}_{1}\in R^{40\times 100},W^{*}_{2}\in R^{100\times 100},W^{*}_{3}\in R^{100\times 10}.$ We choose the $W^{*}_{i}$ to have random independent normal entries with sd=.2. We then initialize the three matrices randomly as $W_{i}(0),i=1,2,3$ to run regular back propagation. For comparison we initialize $W_{i}(0)=0$ and initialize $R_{i}(0)$ randomly. We run the differential equations with $\epsilon=0,.25,.5,1.$, where $\epsilon=0$ corresponds to $FRFB$ and $\epsilon=1$ to URFB. We run 1000 iterations until all 5 algorithms have negligible error. We see the results in figure 12. In the first row, for 3 different runs we show the log-error as a function of iteration, and clearly convergence rate increases with $\epsilon$. In the three rows below that we show the evolution of the correlation of $W_{l}$ and $R_{l}^{t}$ with the same color code. We see that for FRFB (green) the correlation of the weights feeding into the last layer increases to 1 but for the deeper layers that does not hold. Moreover as $\epsilon$ increases to 1 the correlations approach higher values at each layer. The top layer always converges to a correlation very close to 1, lower layers do not reach correlation 1., and interestingly the correlation reached in the input layer is slightly higher than that of the middle layer. Similar non-monotonicity of the correlation was observed in the experiments in figure 11.

6 Discussion

The original idea proposed in [30] of having separate feedback weights undergoing the same Hebbian updates as the feedforward weights yields the original back-propagation algorithm if the feedforward and feedback weights are initialized with the same values. We have shown that even when these weights are initialized differently the performance of the algorithm is comparable to that of back-propagation and significantly outperforms fixed feedback weights as proposed in [17]. The improvement over fixed feedback weights increases with the depth of the network and is demonstrated on challenging benchmarks such as CIFAR10 and CIFAR100. We have also shown that in the long run the feedforward and feedback weights increase their alignment but the performance of the algorithm is comparable to back-propagation even at the initial iterations. We have introduced a cost function whose derivatives lead to local Hebbian updates and provided a proposal for how the associated error signal in the output layer could be implemented in a network. We have shown theoretically, in the linear setting, that adding the update to the feedback weights accelerates the convergence of the error to zero.

These contributions notwithstanding, there are still many aspects of this learning algorithm that are far from biologically plausible. First, although we have removed the need for symmetric connections, we have maintained a symmetric update rule, in that the update of a feedback and feedforward connection connecting two units is the same. To use the formulation in [27] a typical Hebbian update has the form $\Delta W=f(x_{pre})g(x_{post}),$ where $f,g$ are typically not the same function, however in our setting both $f$ and $g$ are linear which yields a symmetric Hebbian update. In [4] it is shown that a mirrored version of STDP could produce this type of symmetric update. Whether this is actually biologically realistic is still an open question.

Another important issue is the timing of the feedforward and feedback weight updates that needs to be very tightly controlled. The update of the feedforward and feedback connections between layer $l$ and $l+1$ requires the feedback signal to layer $l+1$ to have replaced the the feedforward signal in all its units, while the feedforward signal is maintained in layer $l$. This issue is discussed in detail in [7]. They propose a neural model with several compartments. One that receives bottom-up or feedforward input and one that receives top-down feedback input. In a transient phase corresponding to the feedforward processing of the network the top-down input contribution to the neural voltage at the soma is suppressed. Then in a second phase this voltage is allowed in and combined with the feedforward voltage contribution to enable synaptic modifications. In our proposal, instead of combining the two voltages, the top-down voltage would replace the bottom up voltage. Still, in a multilayer network, this would need to be timed in such a way that the previous layer is still responding only to the feedforward input.

An important component of the model proposed in [20] are the synaptic tags that maintain the information on the firing of the pre and post-synaptic neurons allowing for a later synaptic modification based on some reinforcement signal. This may offer a mechanism for controlling the timing of the updates. An alternative direction of research would be to investigate the possibility of desynchronizing the updates, i.e. making the learning process more stochastic. If images of similar classes are shown in sequence it could be that it is not so important when the update occurs, as long as the statistics of the error signal and the feedforward signal are the same.

We have defined the network with neurons that have negative and positive values, and synapses with negative and positive weights. Handling negative weights can be achieved with properly adjusted inhibitory inputs. Handling the negative neural activity is more challenging and it would be of interest to explore an architecture that employs only positive neural activity. Finally we mention the issue of the training protocol. We assume randomly ordered presentation of data from all the classes, many hundreds of times. A more natural protocol would be to learn classes one at a time, perhaps occasionally refreshing the memory of previously learned ones. Because our loss function is local and updates to each class label are independent, one could potentially experiment with alternative protocols and see if they are able to yield similar error rates.

Acknowledgements: This work was supported in part by NIMH award no. R01 MH11555. I’d like to thank Nicolas Brunel, Ulises Pereira and the anonymous referees for helpful comments.

Code: Code for URFB can be found in https://github.com/yaliamit/URFB.git

Appendix 1: Networks for local thresholding

We describe two local networks, one for computing the error signal $\delta_{L}$ (see (6)) at the top level output units, and one to shut off the feedback signal $\delta_{l,i}$ based on (3).

Computing the output error signal

We first describe a network for computing the top level error signal $\delta_{L}$, defined in equation (5), which depends on the activity of the output unit $x_{L}$ as well as on the target signal. There have been many models proposed for Hebbian learning in terms of non-linear functions of both the presynaptic activity and the state of the post-stynaptic neuron, reviewed in detail in [27]. A model for the particular dependence required here, where synaptic modification stops when the input is sufficiently correct was proposed in [5]. A mechanism internal to the neuron is proposed, that shuts off potentiation or depression of its incoming synapses when the input is too high or too low. However, shutting off synaptic modifications, does not manifest itself in the activity of the neuron. An implementation that is internal to the neuron provides no explicit error signal, which can then be propagated to previous layers with the feedback connections. When the network only has two layers, an input and an output, that is not an issue, but with deeper networks we will need to use feedback connections to propagate the error signal.

as above, to avoid complications of modeling excitatory and inhibitory neurons we assume neurons have positive and negative firing rates and synaptic connections are positive and negative. The main idea is to introduce a control unit $t_{c}$ that shuts the main unit $o_{c}$ off when the input is outside the appropriate range.

Let $\delta_{c}$ be the activity of a neuron associated with class $c$ in the top layer, with input given by $h_{c}=\langle W_{c},x\rangle$. For simplicity we omit the index of the input layer. Assume $h_{c}$ always lies in the interval $[-M,M]$. Given a learning threshold $S$ (in the previous section we had $S=1$), we want $\delta_{c}=1$ if class $c$ is presented and $h_{c}<S$, $\delta_{c}=-\mu$ if another class is presented and $h_{c}>-1$ and $\delta_{c}=0$ otherwise. Let $s_{c}$ be the unit providing the supervisory signal: 1 if class $c$ is being presented, -1 otherwise, and let $t_{c}$ be a ‘control’ neuron. The input to $t_{c}$ is simply $o_{c}$ and the full input of $o_{c}$ is

[TABLE]

Also $\delta_{c}=\sigma_{\delta}(H_{c}),\quad t_{c}=\sigma_{t}(o_{c}).$

Let $\mu\leq 1$. The transfer functions of $\delta_{c},t_{c}$ are given by

[TABLE]

and shown in figure 13.

If the supervisory signal is $s_{c}=1$, $t_{c}$ is initialized at [math] and $h_{c}\leq S$ then $M\leq H_{c}\leq 2M+S$, so $\delta_{c}=1$ and $t_{c}=0$. This yeilds the update cycle

[TABLE]

and $\delta_{c}$ is constant at value $1$.

If $h_{c}>S$ then $H_{c}>2M+S$ so that $\delta_{c}=1+\epsilon$ and $t_{c}=1$. Then, $H_{c}=h_{c}\in[-M,M]$ so that $\delta_{c}=0$. This yields the update cycle:

[TABLE]

so that $\delta_{c}$ oscillates between [math] and $1+\epsilon$.

Conversely if the supervisory signal is $s_{c}=-1$ and $h_{c}\geq-S$ then $-2M-S\leq H_{c}\leq-M$ so $\delta_{c}=-\mu$ and $t_{c}=0$ yielding

[TABLE]

and $\delta_{c}$ is constant at $-\mu$. If $h_{c}<-S$ then $H_{c}<-2M-S$, $\delta_{c}=-\mu-\epsilon$ causing $t_{c}=-1$ and $H_{c}=h_{c}$ so that $\delta_{c}=0$ and we get the update cycle:

[TABLE]

so that $\delta_{c}$ oscillates between [math] and $-\mu-\epsilon$.

In summary the activity of $\delta_{c}$ is input dependent. If $sign(\delta_{c})h_{c}<S$ then $\delta_{c}=1,-\mu$ depending on the presented class, and $\Delta W_{ic}=x_{i}\delta_{c}$. If $sign(o)c)h_{c}>S$ then $\delta_{c}$ oscillates and periodically visits the state $\delta_{c}=0$ and no update of synapses connecting to $\delta_{c}$ occurs. Ignoring the oscillation this circuit can be thought of as implementing the rule in (6). The activity of $\delta_{c}$ is precisely the signal that needs to be propagated backwards to the input layer. This network is illustrated in figure 14.

Computing the shutdown of the feedback computation

We describe a local network to compute the expression in equation 3. We add two control neurons $u_{l,i},v_{l,i}$ with a simple threshold activation and input from $x_{l,i}$. We set $u_{l,i}=\bf{1}[{x_{l,i}\geq 1}]$ and $v_{l,i}=\bf{-1}[{x_{l,i}\leq-1}]$. Only one of these units can be active. The input to unit $l,i$ is the top-down feedback $\delta_{l,i}$, and we add $-2Ku_{l,i}+2Kv_{l,i}$. Let

[TABLE]

and set

[TABLE]

Assuming $\delta_{l,i}$ is bounded between $-K,K$ then if either of the units $u_{l,i},v_{l,i}$ are active $\delta_{l,i}-2Ku_{l,i}+2Kv_{l,i}\leq-K$ and $\tilde{\delta}_{l,i}=0$, otherwise $\tilde{\delta}_{l,i}=\delta_{l,i}.$ In terms of timing, before the feedback signal arrives at the unit, but after the value of $x_{l,i}$ is used to update the weights $W_{l}$, the original feedforward activity $x_{l,i}$ activates the units $u_{i,l}$ and $v_{i,l}$ so that their input is added to the incoming feedback signal $\delta_{i,k}$. This sequence of updates is illustrated in figure 15.

Appendix 2: Statement of theorems and proofs

Two layer network

Let $T=U\Lambda_{T}V^{t}$ be the SVD of $T$. Assuming $n_{2}\leq n_{0}$ we have $U\in\mathbb{R}^{n_{2}\times n_{2}},V\in\mathbb{R}^{n_{0}\times n_{2}}$ and $\Lambda_{T}\in\mathbb{R}^{n_{2}\times n_{2}}$ positive diagonal. We set the initial condition for $R_{2}$ by first picking $n_{2}$ orthogonal vectors in $\mathbb{R}^{n_{1}}$ yielding an $n_{1}\times n_{2}$ matrix $S$ with orthogonal columns, and writing $R_{2}(0)=S\Lambda_{R}U^{t}$ with $\Lambda_{R}$ diagonal and positive. This is a restricted initial condition since for the most general initial condition, if we are free to choose $S$, $\Lambda_{R}$ would be upper triangular. Since $W_{1}(0)=W_{2}(0)=0$ and $R_{2}(0)T=S\Lambda_{R}\Lambda_{T}V^{t}$ then we will always have $W_{1}=S\Lambda_{1}V^{t}$ for some $\Lambda_{1}\in\mathbb{R}^{n_{2}\times n_{2}}$ diagonal, and $W_{2}=U\Lambda_{2}S^{t}$ for some $\Lambda_{2}\in\mathbb{R}^{n_{2}\times n_{2}}$ diagonal. This setup yields:

[TABLE]

So that the matrix equations in (5) decouple into $n_{2}$ pairs of scalar ODE’s one for each of the $n_{2}$ columns of $V$.

[TABLE]

We analyze one such pair of scalar equations:

[TABLE]

assuming without loss of generality that $0<\lambda_{T}\leq 1$ and $\lambda_{R}<<\lambda_{T}.$

Theorem 1**.**

Define $e=\lambda_{T}-\lambda_{2}\lambda_{1}$, and assume $\lambda_{1}(0)=0,\lambda_{2}(0)=0$. Then $e^{2}$ converges exponentially fast to 0 and the rate of convergence increases as $\epsilon$ increases.

Proof of Theorem 1.

For $k=2$ equation (Deeper networks) reduces to

[TABLE]

Since $\lambda_{T},\lambda_{R}>0$ and $\lambda_{1}(0)=\lambda_{2}(0)=0$ we see that $\lambda_{1}$ and $\lambda_{2}$ are increasing in time, and so after a finite time are uniformly bounded away from [math]. This implies that the error goes to [math] exponentially fast. We also note that $\lambda_{1}\lambda_{2}=\lambda_{T}$ is a stationary point, so that $\lambda_{1}\lambda_{2}\leq\lambda_{T}$ always holds. If we can show that the factor of $-e^{2}$ increases with $\epsilon$ then the rate of convergence of $e$ to zero increases with $\epsilon.$

Solving for $\lambda_{i}$ in equation (16) and taking the positive solution we can write

[TABLE]

where

[TABLE]

If we show that $\lambda_{1}$ increases with $\epsilon$, then, by (12), $\sqrt{\epsilon}\lambda_{2}=\sqrt{\lambda_{R}^{2}/\epsilon+\lambda_{1}^{2}}-\lambda_{R}/\sqrt{\epsilon},$ is easily seen to increase with $\epsilon$ since $\lambda_{1}$ and $\lambda_{R}$ are positive. (In general, for any positive function $g(\epsilon)$ that is increasing in $\epsilon$ the function $f(\epsilon)=\sqrt{c^{2}/\epsilon+g(\epsilon)}-c/\sqrt{\epsilon}$ is increasing w.r.t to $\epsilon$.) Thus $\epsilon\lambda_{2}^{2}$ is increasing in $\epsilon$ and the factor of $-e^{2}$ in equation (11) is increasing in $\epsilon.$

Since $\lambda_{2}=G(\lambda_{1},\lambda_{R},\epsilon)$ we get the scalar equation for $\lambda_{1}$ as

[TABLE]

Let $h>0$ be a small increment and write,

[TABLE]

Denote $\Delta=\lambda_{\epsilon+h}-\lambda_{\epsilon}$ then to first order

[TABLE]

So if the second term is positive and the initial condition $\Delta(0)=0$ then, using the formula for the solution to first order ODE’s, $\Delta$ is always positive, so that $\lambda_{\epsilon}$ is increasing in $\epsilon$. In our setting by (Proof of Theorem 1.)

[TABLE]

Consequently:

[TABLE]

Thus $\lambda_{1}$ is increasing in $\epsilon.$ ∎

Deeper networks

Let $T=U\Lambda_{T}V^{t}$ be the SVD of $T$. Assuming $n_{k}\leq n_{0},n_{1}\ldots,n_{k-1}$ we have $U\in\mathbb{R}^{n_{k}\times n_{k}},V\in\mathbb{R}^{n_{0}\times n_{k}}$ and $\Lambda_{T}\in\mathbb{R}^{n_{k}\times n_{k}}$ positive diagonal. We set the initial condition for $R_{k}$ by first picking $n_{k}$ orthogonal vectors in $\mathbb{R}^{n_{k-1}}$ yielding an $n_{k-1}\times n_{k}$ matrix $U_{k}$ and then writing $R_{k}(0)=U_{k}\Lambda_{R,k}U^{t}$ with $\Lambda_{R,k}\in\mathbb{R}^{n_{k}\times n_{k}}$ diagonal and positive and $U_{k}\in\mathbb{R}^{n_{k-1}\times n_{k}}$ with orthogonal columns. For general $i$ choose $U_{i}\in\mathbb{R}^{n_{i-1}\times n_{k}}$ orthogonal, $\Lambda_{R,i}\in\mathbb{R}^{n_{k}\times n_{k}}$ positive and diagonal, and set $R_{i}(0)=U_{i}\Lambda_{R,i}U_{i+1}^{t}$. Write $W_{i}=U_{i+1}\Lambda_{i}U_{i}^{t},i=1\ldots,k$ and with $U_{k+1}=U$, and assume $\Lambda_{i}(0)=0$ then the system (5) decouples into $n_{k}$ scalar equations one for each of the directions in $U,V$:

[TABLE]

with $\lambda_{i}(0)=0$ and $\lambda_{R,i}>0$ random. This implies that $\lambda_{i}$ are increasing and $e$ is always positive. Multiplying the $i$’th equation by $(\lambda_{R,i}+\epsilon\lambda_{i})$ and the $i-1$’th equation by $\lambda_{i-1}$ we get the equality

[TABLE]

Since $\lambda_{i}(0)=0$ we can integrate and get

[TABLE]

Rewriting $e=\lambda_{T}-\prod_{i=1}^{k}\lambda_{i},$ we have

[TABLE]

where the third equality follows from (16). It follows that for each $\epsilon$ the error converges exponentially fast to [math] and if it can be shown that $\lambda_{i}$ are increasing in $\epsilon$ the convergence rate is increasing with $\epsilon.$ We are able to show this for $k=3$.

Theorem 2**.**

For $k=3$, assume $\lambda_{1}(0)=0,\lambda_{2}(0)=0$. Assume $\lambda_{R,2},\lambda_{R,3}<\delta<<\lambda_{T}<1$, and assume $\lambda_{R,2}>\frac{1+\sqrt{1+\epsilon}}{2}\lambda_{R,3}$, then $\lambda_{1},\lambda_{2}$ and $\lambda_{3}$ are increasing in $\epsilon$ so that $e$ converges faster to 0 as $\epsilon$ increases.

Proof of Theorem 2.

With $k=3$ equation (Deeper networks) reduces to

[TABLE]

We need to show that $\lambda_{1}$ and $\lambda_{2}$ are increasing in $\epsilon$. The term $\epsilon\lambda_{3}^{2}$ can be handled just like the term $\epsilon\lambda_{2}^{2}$ in the case $k=2$.

Write $H(\lambda_{2})=H(\lambda_{2},\lambda_{R,2},\epsilon)$, $F(\lambda_{2})=F(\lambda_{2},\lambda_{R,3},\epsilon),G(\lambda_{2})=G(\lambda_{2},\lambda_{R,3},\epsilon)$ functions of $\lambda_{2}$. Furthermore set

[TABLE]

all functions of $\lambda_{1}$. Write the equations for $\lambda_{1},\lambda_{2}$ with all other variables eliminated:

[TABLE]

We have

[TABLE]

For $\lambda_{1}$ the derivative of the right hand side of (19) with respect to $\epsilon$ is

[TABLE]

Since $F_{2},G_{2},F_{3},G_{3}>0$ and the derivatives of $G_{2},G_{3}$ with respect to $\epsilon$ are negative the second and third terms are positive. It is left to show that $\frac{dF_{3}}{d\epsilon}>0$. Substituting $G_{2}=\frac{\lambda_{1}^{4}}{(F_{2}+\lambda_{R,2})^{2}}$ in the first term we have:

[TABLE]

For $\lambda_{2}$ the derivative of the right hand side of (19) with respect to $\epsilon$:

[TABLE]

The first and third terms are positive. For the second term we have substituting $G=\frac{\lambda^{2}_{2}}{F+\lambda_{R,3}}$,

[TABLE]

The last factor is

[TABLE]

Assume for small $\delta<1$ that $\lambda_{R,2},\lambda_{R,3}<\delta<<\lambda_{T}$ then an order computation shows that as long as $\lambda_{2}\leq\delta$ then $T_{1}+T_{3}=o(\delta^{2})$ whereas $0>T_{2}=o(\delta^{5})$ and so the sum is positive. When $\lambda_{2}>\lambda_{R,2},\lambda_{R,3}$, then the expression in (LABEL:last) is bounded below by $2\lambda_{R,2}\lambda_{2}-\lambda_{R,3}\lambda_{2}-\lambda_{R,3}\sqrt{1+\epsilon}\lambda_{2}>0$ if $\lambda_{R,2}>\frac{1+\sqrt{1+\epsilon}}{2}\lambda_{R,3}$. Thus $T_{2}>0$. In summary both $\lambda_{1},\lambda_{2}$ are increasing in $\epsilon$ as long as $\lambda_{R,2},\lambda_{R,3}<\delta<<\lambda_{T}$. ∎