A Scalable and Extensible Framework for Superposition-Structured Models

TL;DR

This paper introduces a scalable, extensible proximal quasi-Newton framework for efficiently solving superposition-structured models, improving convergence and applicability to complex structural constraints in statistical learning.

Contribution

The paper develops a novel proximal quasi-Newton framework using smoothed conic dual and LBFGS, enabling efficient optimization of superposition-structured models.

Findings

Achieves super-linear convergence on fused sparse group lasso

Demonstrates scalability on various datasets

Potentially powerful for complex structural models

Abstract

In many learning tasks, structural models usually lead to better interpretability and higher generalization performance. In recent years, however, the simple structural models such as lasso are frequently proved to be insufficient. Accordingly, there has been a lot of work on "superposition-structured" models where multiple structural constraints are imposed. To efficiently solve these "superposition-structured" statistical models, we develop a framework based on a proximal Newton-type method. Employing the smoothed conic dual approach with the LBFGS updating formula, we propose a scalable and extensible proximal quasi-Newton (SEP-QN) framework. Empirical analysis on various datasets shows that our framework is potentially powerful, and achieves super-linear convergence rate for optimizing some popular "superposition-structured" statistical models such as the fused sparse group lasso.