Learning with Structured Sparsity

TL;DR

This paper introduces structured sparsity, a flexible extension of sparsity in learning that leverages arbitrary feature structures, with a theoretical framework, a greedy algorithm, and experimental validation showing improved performance over standard sparsity.

Contribution

It develops a general theory for structured sparsity based on coding complexity, proposes an efficient greedy algorithm, and demonstrates its advantages through experiments.

Findings

Structured sparsity improves learning performance over standard sparsity.

The proposed greedy algorithm effectively approximates coding complexity optimization.

Experiments confirm the practical benefits of structured sparsity in real applications.

Abstract

This paper investigates a new learning formulation called structured sparsity, which is a natural extension of the standard sparsity concept in statistical learning and compressive sensing. By allowing arbitrary structures on the feature set, this concept generalizes the group sparsity idea that has become popular in recent years. A general theory is developed for learning with structured sparsity, based on the notion of coding complexity associated with the structure. It is shown that if the coding complexity of the target signal is small, then one can achieve improved performance by using coding complexity regularization methods, which generalize the standard sparse regularization. Moreover, a structured greedy algorithm is proposed to efficiently solve the structured sparsity problem. It is shown that the greedy algorithm approximately solves the coding complexity optimization problem under appropriate conditions. Experiments are included to demonstrate the advantage of structured sparsity over standard sparsity on some real applications.