Two-stage Ensemble Clustering of Functional Data Using Random Projections

TL;DR

This paper introduces a two-stage ensemble clustering method for functional data using Gaussian process-based random projections, improving accuracy over existing techniques.

Contribution

It presents a novel two-stage clustering framework that combines random projections and data-driven directions for functional data analysis.

Findings

Achieves high clustering accuracy in simulations and real data.

Outperforms many state-of-the-art functional data clustering methods.

Applicable to irregular and partially observed functional data.

Abstract

We propose a computationally simple framework for clustering functional data based on Gaussian-process-generated random projections. In this approach, each curve is first projected onto a large collection of independent Gaussian process realizations. The resulting high-dimensional representations are clustered using the Mean Absolute Difference of Distances (MADD), a dissimilarity measure well suited for high-dimensional settings. A population-level analysis of this dissimilarity provides insight into how random projections help capture distributional differences between functional populations. We introduce a second stage of clustering to additionally leverage on data-driven projection directions. Thus, in Stage I, an initial clustering is obtained using a set of prespecified projection families. In Stage II, this partition is refined by constructing Gaussian random projections based on an estimated covariance operator that uses the first stage of cluster labels. Finally, a normalized cost function is used to select the optimal clustering among candidate solutions. The proposed clustering algorithm is broadly applicable to diverse functional data regimes including irregular and partially observed data. Through extensive simulations and real-data applications, we show that the proposed method achieves a high degree of accuracy and outperforms many of the state-of-the-art methods across a wide range of functional data settings.