Forward-Backward Greedy Algorithms for General Convex Smooth Functions over A Cardinality Constraint

TL;DR

This paper analyzes and improves the theoretical understanding of forward-backward greedy algorithms for sparse feature selection with convex smooth functions, demonstrating their efficiency and effectiveness in practical sensor selection tasks.

Contribution

The paper provides new theoretical bounds for FoBa-obj, shows FoBa-gdt achieves similar performance under restricted strong convexity, and applies these algorithms to sensor selection with superior results.

Findings

FoBa-gdt matches FoBa-obj performance under certain conditions

New bounds improve understanding of greedy algorithms for convex functions

FoBa-gdt outperforms other methods in sensor selection tasks

Abstract

We consider forward-backward greedy algorithms for solving sparse feature selection problems with general convex smooth functions. A state-of-the-art greedy method, the Forward-Backward greedy algorithm (FoBa-obj) requires to solve a large number of optimization problems, thus it is not scalable for large-size problems. The FoBa-gdt algorithm, which uses the gradient information for feature selection at each forward iteration, significantly improves the efficiency of FoBa-obj. In this paper, we systematically analyze the theoretical properties of both forward-backward greedy algorithms. Our main contributions are: 1) We derive better theoretical bounds than existing analyses regarding FoBa-obj for general smooth convex functions; 2) We show that FoBa-gdt achieves the same theoretical performance as FoBa-obj under the same condition: restricted strong convexity condition. Our new bounds are consistent with the bounds of a special case (least squares) and fills a previously existing theoretical gap for general convex smooth functions; 3) We show that the restricted strong convexity condition is satisfied if the number of independent samples is more than $\bar{k}\log d$ where $\bar{k}$ is the sparsity number and $d$ is the dimension of the variable; 4) We apply FoBa-gdt (with the conditional random field objective) to the sensor selection problem for human indoor activity recognition and our results show that FoBa-gdt outperforms other methods (including the ones based on forward greedy selection and L1-regularization).