# Deep learning with asymmetric connections and Hebbian updates

## TL;DR

This paper demonstrates that deep neural networks can be trained effectively using Hebbian updates with asymmetric feedback connections, achieving performance comparable to back-propagation on challenging image datasets, and explores biologically plausible learning rules.

## Contribution

It introduces a Hebbian learning framework with asymmetric, locally updated feedback weights that perform comparably to back-propagation, even with random initializations and untied convolutional layers.

## Key findings

- Hebbian updates achieve similar performance to back-propagation.
- Performance remains robust with random, asymmetric feedback weights.
- Untied convolutional layers perform comparably to tied weights.

## Abstract

We show that deep networks can be trained using Hebbian updates yielding similar performance to ordinary back-propagation on challenging image datasets. To overcome the unrealistic symmetry in connections between layers, implicit in back-propagation, the feedback weights are separate from the feedforward weights. The feedback weights are also updated with a local rule, the same as the feedforward weights - a weight is updated solely based on the product of activity of the units it connects. With fixed feedback weights as proposed in Lillicrap et. al (2016) performance degrades quickly as the depth of the network increases. If the feedforward and feedback weights are initialized with the same values, as proposed in Zipser and Rumelhart (1990), they remain the same throughout training thus precisely implementing back-propagation. We show that even when the weights are initialized differently and at random, and the algorithm is no longer performing back-propagation, performance is comparable on challenging datasets. We also propose a cost function whose derivative can be represented as a local Hebbian update on the last layer. Convolutional layers are updated with tied weights across space, which is not biologically plausible. We show that similar performance is achieved with untied layers, also known as locally connected layers, corresponding to the connectivity implied by the convolutional layers, but where weights are untied and updated separately. In the linear case we show theoretically that the convergence of the error to zero is accelerated by the update of the feedback weights.