On unsupervised projections and second order signals

TL;DR

This paper investigates whether unsupervised linear projections like PCA can preserve second order differences, such as covariance structures, between latent groups in high-dimensional data, with implications for biomedical data analysis.

Contribution

The study provides a theoretical and empirical comparison of PCA and random projections in preserving second order signals in unsupervised high-dimensional data analysis.

Findings

PCA generally outperforms random projections in retaining second order signals.

PCA is often competitive with supervised projections for second order structure.

Projection dimension influences the bias-variance trade-off in signal preservation.

Abstract

Linear projections are widely used in the analysis of high-dimensional data. In unsupervised settings where the data harbour latent classes/clusters, the question of whether class discriminatory signals are retained under projection is crucial. In the case of mean differences between classes, this question has been well studied. However, in many contemporary applications, notably in biomedicine, group differences at the level of covariance or graphical model structure are important. Motivated by such applications, in this paper we ask whether linear projections can preserve differences in second order structure between latent groups. We focus on unsupervised projections, which can be computed without knowledge of class labels. We discuss a simple theoretical framework to study the behaviour of such projections which we use to inform an analysis via quasi-exhaustive enumeration. This allows us to consider the performance, over more than a hundred thousand sets of data-generating population parameters, of two popular projections, namely random projections (RP) and Principal Component Analysis (PCA). Across this broad range of regimes, PCA turns out to be more effective at retaining second order signals than RP and is often even competitive with supervised projection. We complement these results with fully empirical experiments showing 0-1 loss using simulated and real data. We study also the effect of projection dimension, drawing attention to a bias-variance trade-off in this respect. Our results show that PCA can indeed be a suitable first-step for unsupervised analysis, including in cases where differential covariance or graphical model structure are of interest.