A Normative Theory of Adaptive Dimensionality Reduction in Neural Networks

TL;DR

This paper develops biologically plausible online algorithms for adaptive dimensionality reduction in neural networks, allowing the number of output dimensions to vary based on input data eigenspectrum, mimicking biological neural circuits.

Contribution

It introduces three novel adaptive algorithms for dimensionality reduction that dynamically adjust output dimensions based on input covariance eigenspectrum, with biologically plausible neural network implementations.

Findings

Algorithms adapt output dimensions to input eigenspectrum.

Networks model principal neurons and interneurons, reflecting biological circuits.

Algorithms perform effectively in online, real-time settings.

Abstract

To make sense of the world our brains must analyze high-dimensional datasets streamed by our sensory organs. Because such analysis begins with dimensionality reduction, modelling early sensory processing requires biologically plausible online dimensionality reduction algorithms. Recently, we derived such an algorithm, termed similarity matching, from a Multidimensional Scaling (MDS) objective function. However, in the existing algorithm, the number of output dimensions is set a priori by the number of output neurons and cannot be changed. Because the number of informative dimensions in sensory inputs is variable there is a need for adaptive dimensionality reduction. Here, we derive biologically plausible dimensionality reduction algorithms which adapt the number of output dimensions to the eigenspectrum of the input covariance matrix. We formulate three objective functions which, in the offline setting, are optimized by the projections of the input dataset onto its principal subspace scaled by the eigenvalues of the output covariance matrix. In turn, the output eigenvalues are computed as i) soft-thresholded, ii) hard-thresholded, iii) equalized thresholded eigenvalues of the input covariance matrix. In the online setting, we derive the three corresponding adaptive algorithms and map them onto the dynamics of neuronal activity in networks with biologically plausible local learning rules. Remarkably, in the last two networks, neurons are divided into two classes which we identify with principal neurons and interneurons in biological circuits.