Micro-supervised Disturbance Learning: A Perspective of Representation Probability Distribution

TL;DR

This paper introduces a novel micro-supervised disturbance learning framework that enhances representation learning by using minimal label information and small perturbations, improving clustering performance.

Contribution

It proposes the Micro-supervised Disturbance learning approach with new models Micro-DGRBM and Micro-DRBM, utilizing small-perturbation ideology to improve representation distributions with minimal labels.

Findings

Deep Micro-supervised Disturbance Learning outperforms baseline models.

The models effectively use minimal label information to improve clustering.

Contrastive Divergence with SPI enhances representation similarity within clusters.

Abstract

The instability is shown in the existing methods of representation learning based on Euclidean distance under a broad set of conditions. Furthermore, the scarcity and high cost of labels prompt us to explore more expressive representation learning methods which depends on the labels as few as possible. To address these issues, the small-perturbation ideology is firstly introduced on the representation learning model based on the representation probability distribution. The positive small-perturbation information (SPI) which only depend on two labels of each cluster is used to stimulate the representation probability distribution and then two variant models are proposed to fine-tune the expected representation distribution of RBM, namely, Micro-supervised Disturbance GRBM (Micro-DGRBM) and Micro-supervised Disturbance RBM (Micro-DRBM) models. The Kullback-Leibler (KL) divergence of SPI is minimized in the same cluster to promote the representation probability distributions to become more similar in Contrastive Divergence(CD) learning. In contrast, the KL divergence of SPI is maximized in the different clusters to enforce the representation probability distributions to become more dissimilar in CD learning. To explore the representation learning capability under the continuous stimulation of the SPI, we present a deep Micro-supervised Disturbance Learning (Micro-DL) framework based on the Micro-DGRBM and Micro-DRBM models and compare it with a similar deep structure which has not any external stimulation. Experimental results demonstrate that the proposed deep Micro-DL architecture shows better performance in comparison to the baseline method, the most related shallow models and deep frameworks for clustering.