Goodness-of-fit testing of the distribution of posterior classification probabilities for validating model-based clustering

TL;DR

This paper introduces a novel internal goodness-of-fit test for model-based clustering that evaluates the validity of posterior classification probabilities without requiring external labels, suitable for high-dimensional data.

Contribution

It develops a new empirical likelihood-based testing procedure for assessing the distribution of posterior probabilities in clustering, avoiding the curse of dimensionality and external validation.

Findings

The method effectively tests the validity of posterior probabilities in high-dimensional clustering.

It circumvents the curse of dimensionality by focusing on the fixed dimension of the classification simplex.

The test can detect any alternative hypothesis asymptotically using a vector of chi-square-like statistics.

Abstract

We present the first method for assessing the relevance of a model-based clustering result in a general framework. Standard validation criteria, like the adjusted Rand index, rely on external labels to assess partition accuracy; consequently, they are inapplicable to real-world clustering problems where labels are missing. In contrast, our method offers an internal goodness-of-fit diagnostic, since it evaluates the validity of the clustering mechanism by testing the specification of the posterior probabilities of classification defined on the unit simplex. Because this simplex dimension is fixed by the number of clusters, the procedure naturally circumvents the curse of dimensionality, making it applicable to high-dimensional data where traditional density-based tests fail. The testing procedure requires only a consistent estimator of the parameters and the associated posterior classification probabilities for each observation, and its implementation is straightforward, as no additional model fitting is needed. Under the null hypothesis, the method exploits the fact that any functional transformation of the posterior probabilities has the same expectation under both the model being tested and the true data-generating process. The resulting goodness-of-fit test is constructed via an empirical likelihood approach with a growing number of moment conditions, allowing asymptotic detection of any alternative. A block-splitting strategy, employed to account for parameter estimation, provides a vector of test statistics that behave like a vector of independent chi-square random variables. Therefore, the goodness-of-fit of the posterior classification probabilities is assessed via the goodness-of-fit of the vector of empirical likelihood ratio test statistics. Hence, based on the distribution of this vector of statistics, different goodness-of-fit tests (e.g., Kolmogorov-Smirnov) can be used to investigate the distribution of the vector of test statistics with an exact asymptotic significance level.