High-dimensional logistic entropy clustering

TL;DR

This paper provides a theoretical analysis of high-dimensional logistic entropy clustering, showing that entropy minimization can identify mixture separation vectors and recover sparsity support under certain conditions.

Contribution

It is the first to analyze the theoretical properties of entropy-based discriminative clustering with logistic probabilities, especially in high-dimensional sparse settings.

Findings

Entropy minimization identifies mixture separation vectors.

L1-regularization recovers sparse support at standard rates.

Local convexity of the risk ensures fast convergence.

Abstract

Minimization of the (regularized) entropy of classification probabilities is a versatile class of discriminative clustering methods. The classification probabilities are usually defined through the use of some classical losses from supervised classification and the point is to avoid modelisation of the full data distribution by just optimizing the law of the labels conditioned on the observations. We give the first theoretical study of such methods, by specializing to logistic classification probabilities. We prove that if the observations are generated from a two-component isotropic Gaussian mixture, then minimizing the entropy risk over a Euclidean ball indeed allows to identify the separation vector of the mixture. Furthermore, if this separation vector is sparse, then penalizing the empirical risk by a $\ell_{1}$-regularization term allows to infer the separation in a high-dimensional space and to recover its support, at standard rates of sparsity problems. Our approach is based on the local convexity of the logistic entropy risk, that occurs if the separation vector is large enough, with a condition on its norm that is independent from the space dimension. This local convexity property also guarantees fast rates in a classical, low-dimensional setting.