Unsupervised End-to-End Training with a Self-Defined Target

TL;DR

This paper introduces a method that enables end-to-end neural networks to perform high-performance unsupervised learning by adding simple mechanisms, making hardware designed for supervised learning versatile for unlabeled data.

Contribution

The authors propose a novel approach using Winner-Take-All and homeostasis to create self-defined targets, allowing unsupervised training compatible with existing supervised hardware and algorithms.

Findings

Achieved up to 99.2% accuracy on MNIST with unsupervised training.

Demonstrated semi-supervised learning with only 600 labeled MNIST samples.

Extended the method to convolutional layers and equilibrium propagation.

Abstract

Designing algorithms for versatile AI hardware that can learn on the edge using both labeled and unlabeled data is challenging. Deep end-to-end training methods incorporating phases of self-supervised and supervised learning are accurate and adaptable to input data but self-supervised learning requires even more computational and memory resources than supervised learning, too high for current embedded hardware. Conversely, unsupervised layer-by-layer training, such as Hebbian learning, is more compatible with existing hardware but does not integrate well with supervised learning. To address this, we propose a method enabling networks or hardware designed for end-to-end supervised learning to also perform high-performance unsupervised learning by adding two simple elements to the output layer: Winner-Take-All (WTA) selectivity and homeostasis regularization. These mechanisms introduce a "self-defined target" for unlabeled data, allowing purely unsupervised training for both fully-connected and convolutional layers using backpropagation or equilibrium propagation on datasets like MNIST (up to 99.2%), Fashion-MNIST (up to 90.3%), and SVHN (up to 81.5%). We extend this method to semi-supervised learning, adjusting targets based on data type, achieving 96.6% accuracy with only 600 labeled MNIST samples in a multi-layer perceptron. Our results show that this approach can effectively enable networks and hardware initially dedicated to supervised learning to also perform unsupervised learning, adapting to varying availability of labeled data.