A Fast Algorithm for Separated Sparsity via Perturbed Lagrangians

TL;DR

This paper introduces a nearly-linear time algorithm for projecting onto separated sparsity constraints, significantly improving computational efficiency while guaranteeing global optimality, especially useful for structured sparsity in time series data.

Contribution

The paper presents a perturbed Lagrangian relaxation method that computes exact projections onto separated sparsity sets in nearly-linear time, surpassing previous quadratic-time algorithms.

Findings

Achieves nearly-linear time complexity for separated sparsity projection

Provides a globally optimal solution despite non-convex constraints

Demonstrates 10x speed-up in experiments on moderate-sized inputs

Abstract

Sparsity-based methods are widely used in machine learning, statistics, and signal processing. There is now a rich class of structured sparsity approaches that expand the modeling power of the sparsity paradigm and incorporate constraints such as group sparsity, graph sparsity, or hierarchical sparsity. While these sparsity models offer improved sample complexity and better interpretability, the improvements come at a computational cost: it is often challenging to optimize over the (non-convex) constraint sets that capture various sparsity structures. In this paper, we make progress in this direction in the context of separated sparsity -- a fundamental sparsity notion that captures exclusion constraints in linearly ordered data such as time series. While prior algorithms for computing a projection onto this constraint set required quadratic time, we provide a perturbed Lagrangian relaxation approach that computes provably exact projection in only nearly-linear time. Although the sparsity constraint is non-convex, our perturbed Lagrangian approach is still guaranteed to find a globally optimal solution. In experiments, our new algorithms offer a 10$\times$ speed-up already on moderately-size inputs.