When do stepwise algorithms meet subset selection criteria?

TL;DR

This paper investigates when stepwise algorithms can effectively solve subset selection problems by identifying conditions where they align with criteria like AIC or BIC, bridging combinatorial NP-hard problems and greedy methods.

Contribution

It establishes conditions under which stepwise algorithms can find solutions to NP-hard subset selection problems, connecting homotopy methods with classical criteria.

Findings

Identifies conditions for stepwise algorithms to solve subset selection.

Provides theoretical analysis linking greedy algorithms with selection criteria.

Includes an example related to least angle regression.

Abstract

Recent results in homotopy and solution paths demonstrate that certain well-designed greedy algorithms, with a range of values of the algorithmic parameter, can provide solution paths to a sequence of convex optimization problems. On the other hand, in regression many existing criteria in subset selection (including $C_p$, AIC, BIC, MDL, RIC, etc.) involve optimizing an objective function that contains a counting measure. The two optimization problems are formulated as (P1) and (P0) in the present paper. The latter is generally combinatoric and has been proven to be NP-hard. We study the conditions under which the two optimization problems have common solutions. Hence, in these situations a stepwise algorithm can be used to solve the seemingly unsolvable problem. Our main result is motivated by recent work in sparse representation, while two others emerge from different angles: a direct analysis of sufficiency and necessity and a condition on the mostly correlated covariates. An extreme example connected with least angle regression is of independent interest.