Gradient-based Competitive Learning: Theory

TL;DR

This paper introduces a novel approach combining gradient-based unsupervised learning with competitive learning to better replicate input data topology, especially effective for high-dimensional datasets and adaptable to various topological tasks.

Contribution

It presents a new theoretical framework and dual competitive layer that enhances topological learning by working on the transposed input matrix, with proven equivalence and improved handling of high-dimensional data.

Findings

Dual competitive layer is theoretically and experimentally equivalent to the vanilla layer.

The dual layer performs better on very high-dimensional datasets.

The approach can be extended to various topological learning tasks and integrated into complex architectures.

Abstract

Deep learning has been widely used for supervised learning and classification/regression problems. Recently, a novel area of research has applied this paradigm to unsupervised tasks; indeed, a gradient-based approach extracts, efficiently and autonomously, the relevant features for handling input data. However, state-of-the-art techniques focus mostly on algorithmic efficiency and accuracy rather than mimic the input manifold. On the contrary, competitive learning is a powerful tool for replicating the input distribution topology. This paper introduces a novel perspective in this area by combining these two techniques: unsupervised gradient-based and competitive learning. The theory is based on the intuition that neural networks are able to learn topological structures by working directly on the transpose of the input matrix. At this purpose, the vanilla competitive layer and its dual are presented. The former is just an adaptation of a standard competitive layer for deep clustering, while the latter is trained on the transposed matrix. Their equivalence is extensively proven both theoretically and experimentally. However, the dual layer is better suited for handling very high-dimensional datasets. The proposed approach has a great potential as it can be generalized to a vast selection of topological learning tasks, such as non-stationary and hierarchical clustering; furthermore, it can also be integrated within more complex architectures such as autoencoders and generative adversarial networks.