Convex Relaxation for Combinatorial Penalties

TL;DR

This paper introduces a convex relaxation framework for structured sparsity-inducing norms, unifying various approaches and establishing theoretical links with submodular functions and latent representations.

Contribution

It proposes a unifying convex relaxation approach for combinatorial penalties, characterizes the relaxation's tightness, and connects it to existing norm-based methods.

Findings

Convex relaxations effectively approximate combinatorial penalties.

The lower combinatorial envelope characterizes relaxation tightness.

Links established with latent group Lasso and submodular functions.

Abstract

In this paper, we propose an unifying view of several recently proposed structured sparsity-inducing norms. We consider the situation of a model simultaneously (a) penalized by a set- function de ned on the support of the unknown parameter vector which represents prior knowledge on supports, and (b) regularized in Lp-norm. We show that the natural combinatorial optimization problems obtained may be relaxed into convex optimization problems and introduce a notion, the lower combinatorial envelope of a set-function, that characterizes the tightness of our relaxations. We moreover establish links with norms based on latent representations including the latent group Lasso and block-coding, and with norms obtained from submodular functions.