A Successive Two-stage Method for Sparse Generalized Eigenvalue Problems

TL;DR

This paper introduces a novel successive two-stage algorithm for solving the NP-hard sparse generalized eigenvalue problem, effectively avoiding local optima and improving solution quality through iterative support adjustment and convergence analysis.

Contribution

The paper proposes a new iterative two-stage method with support alteration for sGEP, enhancing solution quality and convergence over existing algorithms.

Findings

Significant improvement in objective value over existing methods

Effective avoidance of local optima in sGEP solutions

Demonstrated convergence properties of the proposed method

Abstract

The Sparse Generalized Eigenvalue Problem (sGEP), a pervasive challenge in statistical learning methods including sparse principal component analysis, sparse Fisher's discriminant analysis, and sparse canonical correlation analysis, presents significant computational complexity due to its NP-hardness. The primary aim of sGEP is to derive a sparse vector approximation of the largest generalized eigenvector, effectively posing this as a sparse optimization problem. Conventional algorithms for sGEP, however, often succumb to local optima and exhibit significant dependency on initial points. This predicament necessitates a more refined approach to avoid local optima and achieve an improved solution in terms of sGEP's objective value, which we address in this paper through a novel successive two-stage method. The first stage of this method incorporates an algorithm for sGEP capable of yielding a stationary point from any initial point. The subsequent stage refines this stationary point by adjusting its support, resulting in a point with an enhanced objective value relative to the original stationary point. This support adjustment is achieved through a novel procedure we have named support alteration. The final point derived from the second stage then serves as the initial point for the algorithm in the first stage, creating a cyclical process that continues until a predetermined stopping criterion is satisfied. We also provide a comprehensive convergence analysis of this process. Through extensive experimentation under various settings, our method has demonstrated significant improvements in the objective value of sGEP compared to existing methodologies, underscoring its potential as a valuable tool in statistical learning and optimization.