Deflation-Free Optimal Scoring

TL;DR

This paper introduces DFSOS, a deflation-free method for sparse discriminant analysis that estimates all discriminant vectors simultaneously, improving accuracy and robustness in high-dimensional settings.

Contribution

The paper proposes a novel deflation-free approach combining Bregman iteration and orthogonality constraints for sparse optimal scoring, addressing limitations of sequential methods.

Findings

DFSOS achieves comparable or better classification accuracy than existing methods.

Extensive experiments validate the robustness and effectiveness of DFSOS in high-dimensional data.

The method converges to stationary points under mild conditions.

Abstract

Sparse Optimal Scoring (SOS) reformulates linear discriminant analysis to enable feature selection through elastic net regularization, making it well-suited for high-dimensional settings where the number of features exceeds observations. Most existing SOS methods use deflation-based strategies that compute discriminant vectors sequentially, which can propagate errors and produce suboptimal solutions. We propose a novel approach that estimates all discriminant vectors simultaneously under an explicit global orthogonality constraint, which we call Deflation-Free Sparse Optimal Scoring (DFSOS). DFSOS combines Bregman iteration with orthogonality-constrained optimization, decomposing the problem into tractable subproblems for scoring vectors, discriminant vectors, and orthogonality enforcement. We establish convergence to stationary points of the augmented Lagrangian under mild conditions. Extensive experiments using synthetic data and real-world time series data demonstrate that DFSOS achieves classification accuracy comparable to or better than existing deflation-based methods. These results indicate that deflation-free approaches offer a robust and effective framework for sparse discriminant analysis in high-dimensional problems.