A Subspace Acceleration Method for Minimization Involving a Group Sparsity-Inducing Regularizer

TL;DR

This paper introduces a novel subspace acceleration method for minimizing functions with group sparsity regularizers, improving convergence and efficiency in machine learning tasks like logistic regression.

Contribution

The paper proposes a new optimization framework combining subspace acceleration, domain decomposition, and support identification for group sparsity problems, with proven convergence properties.

Findings

Global convergence to an $\e$-approximate solution within $O(\e^{-(1+p)})$ iterations.

Superlinear local convergence demonstrated.

Preliminary results show improved efficiency and robustness over existing methods.

Abstract

We consider the problem of minimizing an objective function that is the sum of a convex function and a group sparsity-inducing regularizer. Problems that integrate such regularizers arise in modern machine learning applications, often for the purpose of obtaining models that are easier to interpret and that have higher predictive accuracy. We present a new method for solving such problems that utilize subspace acceleration, domain decomposition, and support identification. Our analysis shows, under common assumptions, that the iterate sequence generated by our framework is globally convergent, converges to an $\epsilon$-approximate solution in at most $O(\epsilon^{-(1+p)})$ (respectively, $O(\epsilon^{-(2+p)})$) iterations for all $\epsilon$ bounded above and large enough (respectively, all $\epsilon$ bounded above) where $p > 0$ is an algorithm parameter, and exhibits superlinear local convergence. Preliminary numerical results for the task of binary classification based on regularized logistic regression show that our approach is efficient and robust, with the ability to outperform a state-of-the-art method.