Kernel similarity matching with Hebbian neural networks

TL;DR

This paper introduces a biologically plausible neural network method for kernel similarity matching that learns nonlinear features directly, producing sparse, pattern-selective representations and outperforming some existing approximation techniques in certain settings.

Contribution

It proposes a new online correlation-based learning algorithm for kernel similarity matching that directly learns nonlinear features with a single-layer recurrent neural network.

Findings

Comparable or better than Nyström methods at low output dimensions

Produces sparse and pattern-selective representations

Less accurate than Nyström at very high output dimensions

Abstract

Recent works have derived neural networks with online correlation-based learning rules to perform \textit{kernel similarity matching}. These works applied existing linear similarity matching algorithms to nonlinear features generated with random Fourier methods. In this paper attempt to perform kernel similarity matching by directly learning the nonlinear features. Our algorithm proceeds by deriving and then minimizing an upper bound for the sum of squared errors between output and input kernel similarities. The construction of our upper bound leads to online correlation-based learning rules which can be implemented with a 1 layer recurrent neural network. In addition to generating high-dimensional linearly separable representations, we show that our upper bound naturally yields representations which are sparse and selective for specific input patterns. We compare the approximation quality of our method to neural random Fourier method and variants of the popular but non-biological "Nystr{\"o}m" method for approximating the kernel matrix. Our method appears to be comparable or better than randomly sampled Nystr{\"o}m methods when the outputs are relatively low dimensional (although still potentially higher dimensional than the inputs) but less faithful when the outputs are very high dimensional.