Worst-Case Linear Discriminant Analysis as Scalable Semidefinite Feasibility Problems

TL;DR

This paper introduces a scalable semidefinite programming approach for worst-case linear discriminant analysis, improving robustness and computational efficiency over traditional methods, and demonstrating superior classification performance.

Contribution

It reformulates WLDA into a sequence of semidefinite feasibility problems and develops a fast quasi-Newton based solver, significantly enhancing scalability and speed.

Findings

Achieves better classification performance than standard LDA.

Much faster and more scalable than interior-point SDP solvers.

Reduces computational complexity from cubic to cubic in matrix size.

Abstract

In this paper, we propose an efficient semidefinite programming (SDP) approach to worst-case linear discriminant analysis (WLDA). Compared with the traditional LDA, WLDA considers the dimensionality reduction problem from the worst-case viewpoint, which is in general more robust for classification. However, the original problem of WLDA is non-convex and difficult to optimize. In this paper, we reformulate the optimization problem of WLDA into a sequence of semidefinite feasibility problems. To efficiently solve the semidefinite feasibility problems, we design a new scalable optimization method with quasi-Newton methods and eigen-decomposition being the core components. The proposed method is orders of magnitude faster than standard interior-point based SDP solvers. Experiments on a variety of classification problems demonstrate that our approach achieves better performance than standard LDA. Our method is also much faster and more scalable than standard interior-point SDP solvers based WLDA. The computational complexity for an SDP with $m$ constraints and matrices of size $d$ by $d$ is roughly reduced from $\mathcal{O}(m^3+md^3+m^2d^2)$ to $\mathcal{O}(d^3)$ ($m>d$ in our case).