Unlocking the Potential of Similarity Matching: Scalability, Supervision and Pre-training

TL;DR

This paper advances similarity matching algorithms by enhancing scalability, introducing supervised objectives, and leveraging pre-training, bridging biological plausibility with computational efficiency.

Contribution

It proposes scalable convolutional similarity matching, a supervised variant for stacking, and demonstrates effective pre-training of neural architectures.

Findings

Scalable convolutional similarity matching implemented in PyTorch.

Supervised similarity matching for layer stacking.

Pre-trained models with similarity matching outperform or match backpropagation in feature evaluation.

Abstract

While effective, the backpropagation (BP) algorithm exhibits limitations in terms of biological plausibility, computational cost, and suitability for online learning. As a result, there has been a growing interest in developing alternative biologically plausible learning approaches that rely on local learning rules. This study focuses on the primarily unsupervised similarity matching (SM) framework, which aligns with observed mechanisms in biological systems and offers online, localized, and biologically plausible algorithms. i) To scale SM to large datasets, we propose an implementation of Convolutional Nonnegative SM using PyTorch. ii) We introduce a localized supervised SM objective reminiscent of canonical correlation analysis, facilitating stacking SM layers. iii) We leverage the PyTorch implementation for pre-training architectures such as LeNet and compare the evaluation of features against BP-trained models. This work combines biologically plausible algorithms with computational efficiency opening multiple avenues for further explorations.