Discriminative Entropy Clustering and its Relation to K-means and SVM

TL;DR

This paper explores entropy-based clustering, clarifies its relation to K-means and SVM, and introduces a new self-labeling method that enhances deep clustering performance.

Contribution

It provides theoretical insights into entropy clustering, disproves some prior claims, and proposes a novel self-labeling approach with an EM algorithm for improved deep clustering.

Findings

Theoretical relation between entropy clustering and SVM margin maximization.

Disproof of some earlier claims about entropy clustering properties.

State-of-the-art performance on deep clustering benchmarks.

Abstract

Maximization of mutual information between the model's input and output is formally related to "decisiveness" and "fairness" of the softmax predictions, motivating these unsupervised entropy-based criteria for clustering. First, in the context of linear softmax models, we discuss some general properties of entropy-based clustering. Disproving some earlier claims, we point out fundamental differences with K-means. On the other hand, we prove the margin maximizing property for decisiveness establishing a relation to SVM-based clustering. Second, we propose a new self-labeling formulation of entropy clustering for general softmax models. The pseudo-labels are introduced as auxiliary variables "splitting" the fairness and decisiveness. The derived self-labeling loss includes the reverse cross-entropy robust to pseudo-label errors and allows an efficient EM solver for pseudo-labels. Our algorithm improves the state of the art on several standard benchmarks for deep clustering.