Augmenting Supervised Learning by Meta-learning Unsupervised Local Rules

TL;DR

This paper introduces HAT, a meta-learning algorithm that combines supervised and unsupervised learning rules, demonstrating improved performance on Fashion-MNIST and revealing insights into the learned update rules.

Contribution

The paper proposes a novel meta-learning approach to combine gradient-based supervised learning with learned Hebbian-like unsupervised rules, enhancing model performance.

Findings

HAT outperforms supervised learning alone on Fashion-MNIST.

Unsupervised synaptic activity provides a valuable augmentation signal.

Meta-learned rules tend to become non-Hebbian over time.

Abstract

The brain performs unsupervised learning and (perhaps) simultaneous supervised learning. This raises the question as to whether a hybrid of supervised and unsupervised methods will produce better learning. Inspired by the rich space of Hebbian learning rules, we set out to directly learn the unsupervised learning rule on local information that best augments a supervised signal. We present the Hebbian-augmented training algorithm (HAT) for combining gradient-based learning with an unsupervised rule on pre-synpatic activity, post-synaptic activities, and current weights. We test HAT's effect on a simple problem (Fashion-MNIST) and find consistently higher performance than supervised learning alone. This finding provides empirical evidence that unsupervised learning on synaptic activities provides a strong signal that can be used to augment gradient-based methods. We further find that the meta-learned update rule is a time-varying function; thus, it is difficult to pinpoint an interpretable Hebbian update rule that aids in training. We do find that the meta-learner eventually degenerates into a non-Hebbian rule that preserves important weights so as not to disturb the learner's convergence.