Regression based thresholds in principal loading analysis

TL;DR

This paper explores the relationship between principal loading analysis and multivariate linear regression, providing conditions for their variable selection to coincide and extending threshold choices for principal loading analysis.

Contribution

It establishes conditions under which principal loading analysis and regression-based variable selection agree and offers a new approach for choosing thresholds in principal loading analysis.

Findings

Conditions for shared variable selection between methods

Extended threshold selection criteria for principal loading analysis

Theoretical insights linking PCA-based and regression-based variable selection

Abstract

Principal loading analysis is a dimension reduction method that discards variables which have only a small distorting effect on the covariance matrix. As a special case, principal loading analysis discards variables that are not correlated with the remaining ones. In multivariate linear regression on the other hand, predictors that are neither correlated with both the remaining predictors nor with the dependent variables have a regression coefficients equal to zero. Hence, if the goal is to select a number of predictors, variables that do not correlate are discarded as it is also done in principal loading analysis. That both methods select the same variables occurs not only for the special case of zero correlation however. We contribute conditions under which both methods share the same variable selection. Further, we extend those conditions to provide a choice for the threshold in principal loading analysis which only follows recommendations based on simulation results so far.