Structured Sparsity via Alternating Direction Methods

TL;DR

This paper introduces a unified optimization framework using augmented Lagrangian methods and alternating partial-linearization for structured sparsity problems, effectively handling non-smooth regularizers like overlapping Group Lasso and $l_1/l_$-norms.

Contribution

It develops new algorithms with accelerated convergence for structured sparsity regularization, applicable to high-dimensional feature selection with prior group knowledge.

Findings

Algorithms require $O(1/\sqrt{\epsilon})$ iterations for $\\epsilon$-optimal solutions.

Demonstrated efficiency on multiple datasets and real-world applications.

Compared the effectiveness of overlapping Group Lasso and $l_1/l_\infty$-norm regularizations.

Abstract

We consider a class of sparse learning problems in high dimensional feature space regularized by a structured sparsity-inducing norm which incorporates prior knowledge of the group structure of the features. Such problems often pose a considerable challenge to optimization algorithms due to the non-smoothness and non-separability of the regularization term. In this paper, we focus on two commonly adopted sparsity-inducing regularization terms, the overlapping Group Lasso penalty $l_1/l_2$-norm and the $l_1/l_\infty$-norm. We propose a unified framework based on the augmented Lagrangian method, under which problems with both types of regularization and their variants can be efficiently solved. As the core building-block of this framework, we develop new algorithms using an alternating partial-linearization/splitting technique, and we prove that the accelerated versions of these algorithms require $O(\frac{1}{\sqrt{\epsilon}})$ iterations to obtain an $\epsilon$-optimal solution. To demonstrate the efficiency and relevance of our algorithms, we test them on a collection of data sets and apply them to two real-world problems to compare the relative merits of the two norms.