Penalized versus constrained generalized eigenvalue problems

TL;DR

This paper compares $ abla_1$ penalties and constraints in generalized eigenvalue problems, showing constraints often yield sparser solutions and better variable selection, supported by empirical and theoretical evidence.

Contribution

It demonstrates that $ abla_1$ penalties may not produce sparse solutions, whereas $ abla_1$ constraints improve sparsity and variable selection in eigenvalue problems.

Findings

$ abla_1$ penalties can fail to produce sparse solutions

$ abla_1$ constraints enhance sparsity and variable selection

Empirical and theoretical support for constraints over penalties

Abstract

We investigate the difference between using an $\ell_1$ penalty versus an $\ell_1$ constraint in generalized eigenvalue problems, such as principal component analysis and discriminant analysis. Our main finding is that an $\ell_1$ penalty may fail to provide very sparse solutions; a severe disadvantage for variable selection that can be remedied by using an $\ell_1$ constraint. Our claims are supported both by empirical evidence and theoretical analysis. Finally, we illustrate the advantages of an $\ell_1$ constraint in the context of discriminant analysis and principal component analysis.