AgFlow: Fast Model Selection of Penalized PCA via Implicit Regularization Effects of Gradient Flow

TL;DR

AgFlow is a fast model selection method for penalized PCA that leverages implicit regularization effects of gradient flow to efficiently compute solution paths, outperforming existing methods in computational speed.

Contribution

This paper introduces AgFlow, a novel approach that significantly reduces computation time for penalized PCA model selection by utilizing implicit regularization effects of gradient flow.

Findings

AgFlow outperforms existing methods in computation speed.

AgFlow accurately computes the solution path of L2-penalized PCA.

Extensive experiments validate AgFlow's efficiency on real-world datasets.

Abstract

Principal component analysis (PCA) has been widely used as an effective technique for feature extraction and dimension reduction. In the High Dimension Low Sample Size (HDLSS) setting, one may prefer modified principal components, with penalized loadings, and automated penalty selection by implementing model selection among these different models with varying penalties. The earlier work [1, 2] has proposed penalized PCA, indicating the feasibility of model selection in $L_2$- penalized PCA through the solution path of Ridge regression, however, it is extremely time-consuming because of the intensive calculation of matrix inverse. In this paper, we propose a fast model selection method for penalized PCA, named Approximated Gradient Flow (AgFlow), which lowers the computation complexity through incorporating the implicit regularization effect introduced by (stochastic) gradient flow [3, 4] and obtains the complete solution path of $L_2$-penalized PCA under varying $L_2$-regularization. We perform extensive experiments on real-world datasets. AgFlow outperforms existing methods (Oja [5], Power [6], and Shamir [7] and the vanilla Ridge estimators) in terms of computation costs.