Untangling Lariats: Subgradient Following of Variationally Penalized Objectives

TL;DR

This paper introduces a unified framework for subgradient optimization of convex problems with variational penalties, encompassing known algorithms and enabling new high-order filtering methods for temporal sequences.

Contribution

It presents a novel lattice-based subgradient method for variational penalties and extends the approach to multivariate problems with group sparsity.

Findings

Unified framework for variational penalties including fused lasso and isotonic regression.

Efficient solvers for high-order filtering problems in temporal sequences.

Extension to multivariate problems promoting group sparsity.

Abstract

We describe an apparatus for subgradient-following of the optimum of convex problems with variational penalties. In this setting, we receive a sequence $y_i,\ldots,y_n$ and seek a smooth sequence $x_1,\ldots,x_n$. The smooth sequence needs to attain the minimum Bregman divergence to an input sequence with additive variational penalties in the general form of $\sum_i{}g_i(x_{i+1}-x_i)$. We derive known algorithms such as the fused lasso and isotonic regression as special cases of our approach. Our approach also facilitates new variational penalties such as non-smooth barrier functions. We then derive a novel lattice-based procedure for subgradient following of variational penalties characterized through the output of arbitrary convolutional filters. This paradigm yields efficient solvers for high-order filtering problems of temporal sequences in which sparse discrete derivatives such as acceleration and jerk are desirable. We also introduce and analyze new multivariate problems in which $\mathbf{x}_i,\mathbf{y}_i\in\mathbb{R}^d$ with variational penalties that depend on $\|\mathbf{x}_{i+1}-\mathbf{x}_i\|$. The norms we consider are $\ell_2$ and $\ell_\infty$ which promote group sparsity.