Self-calibrating Neural Networks for Dimensionality Reduction

TL;DR

This paper introduces self-calibrating online algorithms for neural network-based dimensionality reduction that adaptively determine the number of output dimensions without prior knowledge of singular values, using novel regularizers.

Contribution

It presents new online algorithms with self-calibrating thresholds derived from similarity matching, enabling adaptive dimensionality reduction in streaming data.

Findings

Algorithms effectively determine output dimensions adaptively.

Mathematical analysis confirms convergence and stability.

Simulations demonstrate practical effectiveness in various settings.

Abstract

Recently, a novel family of biologically plausible online algorithms for reducing the dimensionality of streaming data has been derived from the similarity matching principle. In these algorithms, the number of output dimensions can be determined adaptively by thresholding the singular values of the input data matrix. However, setting such threshold requires knowing the magnitude of the desired singular values in advance. Here we propose online algorithms where the threshold is self-calibrating based on the singular values computed from the existing observations. To derive these algorithms from the similarity matching cost function we propose novel regularizers. As before, these online algorithms can be implemented by Hebbian/anti-Hebbian neural networks in which the learning rule depends on the chosen regularizer. We demonstrate both mathematically and via simulation the effectiveness of these online algorithms in various settings.