Bayesian dimensionality reduction with PCA using penalized semi-integrated likelihood

TL;DR

This paper introduces PESEL, a Bayesian model selection criterion for PCA that adapts to data set size, demonstrating robustness and outperforming existing methods through extensive simulations and real data analysis.

Contribution

The paper develops a new Bayesian criterion, PESEL, for estimating principal components, unifying and extending existing approaches with a framework suitable for large variable-to-observation ratios.

Findings

PESEL criteria outperform state-of-the-art methods in simulations.

PESEL shows robustness against deviations from probabilistic assumptions.

The method is implemented in the R package varclust.

Abstract

We discuss the problem of estimating the number of principal components in Principal Com- ponents Analysis (PCA). Despite of the importance of the problem and the multitude of solutions proposed in the literature, it comes as a surprise that there does not exist a coherent asymptotic framework which would justify different approaches depending on the actual size of the data set. In this paper we address this issue by presenting an approximate Bayesian approach based on Laplace approximation and introducing a general method for building the model selection criteria, called PEnalized SEmi-integrated Likelihood (PESEL). Our general framework encompasses a variety of existing approaches based on probabilistic models, like e.g. Bayesian Information Criterion for the Probabilistic PCA (PPCA), and allows for construction of new criteria, depending on the size of the data set at hand. Specifically, we define PESEL when the number of variables substantially exceeds the number of observations. We also report results of extensive simulation studies and real data analysis, which illustrate good properties of our proposed criteria as compared to the state-of- the-art methods and very recent proposals. Specifially, these simulations show that PESEL based criteria can be quite robust against deviations from the probabilistic model assumptions. Selected PESEL based criteria for the estimation of the number of principal components are implemented in R package varclust, which is available on github (https://github.com/psobczyk/varclust).