An Information-Theoretic Framework for Fast and Robust Unsupervised Learning via Neural Population Infomax

TL;DR

This paper introduces a novel information-theoretic framework for fast, robust unsupervised learning of neural representations, leveraging an asymptotic approximation and hierarchical infomax to improve efficiency and robustness.

Contribution

It proposes a hierarchical infomax method with an efficient gradient descent algorithm for large-scale neural data, enhancing training speed and robustness over existing methods.

Findings

The method is robust and efficient in extracting features from datasets.

It outperforms existing methods in training speed and robustness.

Easily extendable to deep neural network training.

Abstract

A framework is presented for unsupervised learning of representations based on infomax principle for large-scale neural populations. We use an asymptotic approximation to the Shannon's mutual information for a large neural population to demonstrate that a good initial approximation to the global information-theoretic optimum can be obtained by a hierarchical infomax method. Starting from the initial solution, an efficient algorithm based on gradient descent of the final objective function is proposed to learn representations from the input datasets, and the method works for complete, overcomplete, and undercomplete bases. As confirmed by numerical experiments, our method is robust and highly efficient for extracting salient features from input datasets. Compared with the main existing methods, our algorithm has a distinct advantage in both the training speed and the robustness of unsupervised representation learning. Furthermore, the proposed method is easily extended to the supervised or unsupervised model for training deep structure networks.