A Block Decomposition Algorithm for Sparse Optimization

TL;DR

This paper introduces a novel block decomposition algorithm for sparse optimization that combines combinatorial search and coordinate descent, achieving better solutions and convergence rates in large-scale problems.

Contribution

The paper proposes a new block decomposition algorithm that integrates combinatorial search with coordinate descent, improving solution quality and convergence in sparse optimization.

Findings

Achieves stronger stationary points than existing coordinate-wise methods.

Establishes convergence rate for the proposed algorithm.

Outperforms greedy pursuit methods in accuracy on benchmark problems.

Abstract

Sparse optimization is a central problem in machine learning and computer vision. However, this problem is inherently NP-hard and thus difficult to solve in general. Combinatorial search methods find the global optimal solution but are confined to small-sized problems, while coordinate descent methods are efficient but often suffer from poor local minima. This paper considers a new block decomposition algorithm that combines the effectiveness of combinatorial search methods and the efficiency of coordinate descent methods. Specifically, we consider a random strategy or/and a greedy strategy to select a subset of coordinates as the working set, and then perform a global combinatorial search over the working set based on the original objective function. We show that our method finds stronger stationary points than Amir Beck et al.'s coordinate-wise optimization method. In addition, we establish the convergence rate of our algorithm. Our experiments on solving sparse regularized and sparsity constrained least squares optimization problems demonstrate that our method achieves state-of-the-art performance in terms of accuracy. For example, our method generally outperforms the well-known greedy pursuit method.