Post-clustering Inference under Dependence

TL;DR

This paper extends post-clustering inference methods to handle arbitrary dependence structures in data, broadening their applicability beyond the independent Gaussian case, and demonstrates their effectiveness on synthetic and real datasets.

Contribution

It develops a theoretical framework for post-clustering inference under dependence, applicable to hierarchical clustering and k-means, with conditions for covariance estimation.

Findings

Framework extended to dependent data structures

Conditions established for covariance matrix estimation

Validated on synthetic and protein structure data

Abstract

Recent work by Gao et al. (JASA 2022) has laid the foundations for post-clustering inference, establishing a theoretical framework allowing to test for differences between means of estimated clusters. Additionally, they studied the estimation of unknown parameters while controlling the selective type I error. However, their theory was developed for independent observations identically distributed as $p$-dimensional Gaussian variables, where the parameter estimation could only be performed for spherical covariance matrices. Here, we aim at extending this framework to a more convenient scenario for practical applications, where arbitrary dependence structures between observations and features are allowed. We establish sufficient conditions for extending the setting presented by Gao et al. to the general dependence framework. Moreover, we assess theoretical conditions allowing the compatible estimation of a covariance matrix. The theory is developed for hierarchical agglomerative clustering algorithms with several types of linkages, and for the $k$-means algorithm. We illustrate our method with synthetic data and real data of protein structures.