Hebbian learning with gradients: Hebbian convolutional neural networks with modern deep learning frameworks

TL;DR

This paper demonstrates that Hebbian learning rules can be implemented efficiently within modern deep learning frameworks, enabling the development of biologically plausible convolutional neural networks with improved performance through specific interventions.

Contribution

It introduces a method to implement Hebbian learning rules using gradient-based losses in deep frameworks and applies this to multi-layer networks for object recognition.

Findings

Hebbian rules can be exactly implemented via gradient-based losses.

Interventions lead to sparser features and significantly improved performance.

Higher layers tend to learn simple, large features like Gabor filters.

Abstract

Deep learning networks generally use non-biological learning methods. By contrast, networks based on more biologically plausible learning, such as Hebbian learning, show comparatively poor performance and difficulties of implementation. Here we show that Hebbian learning in hierarchical, convolutional neural networks can be implemented almost trivially with modern deep learning frameworks, by using specific losses whose gradients produce exactly the desired Hebbian updates. We provide expressions whose gradients exactly implement a plain Hebbian rule (dw ~= xy), Grossberg's instar rule (dw ~= y(x-w)), and Oja's rule (dw ~= y(x-yw)). As an application, we build Hebbian convolutional multi-layer networks for object recognition. We observe that higher layers of such networks tend to learn large, simple features (Gabor-like filters and blobs), explaining the previously reported decrease in decoding performance over successive layers. To combat this tendency, we introduce interventions (denser activations with sparse plasticity, pruning of connections between layers) which result in sparser learned features, massively increase performance, and allow information to increase over successive layers. We hypothesize that more advanced techniques (dynamic stimuli, trace learning, feedback connections, etc.), together with the massive computational boost offered by modern deep learning frameworks, could greatly improve the performance and biological relevance of multi-layer Hebbian networks.