FastHebb: Scaling Hebbian Training of Deep Neural Networks to ImageNet Level

TL;DR

FastHebb introduces a scalable Hebbian learning method that significantly accelerates training, enabling its application to large datasets like ImageNet in semi-supervised scenarios, outperforming previous solutions in speed.

Contribution

The paper presents FastHebb, a novel, efficient Hebbian learning algorithm that scales to large datasets by optimizing update computation and leveraging GPU matrix multiplication.

Findings

FastHebb achieves up to 50x faster training speeds.

It is the first Hebbian method successfully applied to ImageNet.

Demonstrates effectiveness in semi-supervised learning scenarios.

Abstract

Learning algorithms for Deep Neural Networks are typically based on supervised end-to-end Stochastic Gradient Descent (SGD) training with error backpropagation (backprop). Backprop algorithms require a large number of labelled training samples to achieve high performance. However, in many realistic applications, even if there is plenty of image samples, very few of them are labelled, and semi-supervised sample-efficient training strategies have to be used. Hebbian learning represents a possible approach towards sample efficient training; however, in current solutions, it does not scale well to large datasets. In this paper, we present FastHebb, an efficient and scalable solution for Hebbian learning which achieves higher efficiency by 1) merging together update computation and aggregation over a batch of inputs, and 2) leveraging efficient matrix multiplication algorithms on GPU. We validate our approach on different computer vision benchmarks, in a semi-supervised learning scenario. FastHebb outperforms previous solutions by up to 50 times in terms of training speed, and notably, for the first time, we are able to bring Hebbian algorithms to ImageNet scale.