Training Convolutional Neural Networks With Hebbian Principal Component Analysis

TL;DR

This paper introduces a nonlinear Hebbian PCA learning rule for training convolutional neural networks, demonstrating improved feature extraction on CIFAR-10 and advancing biologically plausible learning methods.

Contribution

It proposes a novel nonlinear Hebbian PCA rule for CNN training, outperforming previous Hebbian strategies and enhancing biologically plausible learning approaches.

Findings

Improved feature extraction on CIFAR-10.

Enhanced training efficiency compared to previous Hebbian methods.

Motivates further research into biologically plausible algorithms.

Abstract

Recent work has shown that biologically plausible Hebbian learning can be integrated with backpropagation learning (backprop), when training deep convolutional neural networks. In particular, it has been shown that Hebbian learning can be used for training the lower or the higher layers of a neural network. For instance, Hebbian learning is effective for re-training the higher layers of a pre-trained deep neural network, achieving comparable accuracy w.r.t. SGD, while requiring fewer training epochs, suggesting potential applications for transfer learning. In this paper we build on these results and we further improve Hebbian learning in these settings, by using a nonlinear Hebbian Principal Component Analysis (HPCA) learning rule, in place of the Hebbian Winner Takes All (HWTA) strategy used in previous work. We test this approach in the context of computer vision. In particular, the HPCA rule is used to train Convolutional Neural Networks in order to extract relevant features from the CIFAR-10 image dataset. The HPCA variant that we explore further improves the previous results, motivating further interest towards biologically plausible learning algorithms.