Super-Linear Convergence of Dual Augmented-Lagrangian Algorithm for Sparsity Regularized Estimation

TL;DR

This paper proves that the Dual Augmented-Lagrangian (DAL) algorithm converges super-linearly for sparsity-regularized estimation problems, with milder assumptions and broader applicability, supported by theoretical analysis and extensive experiments.

Contribution

It provides a new interpretation of DAL as a proximal minimization method, enabling super-linear convergence analysis under milder conditions and extending its applicability to various sparse estimation problems.

Findings

DAL converges super-linearly under certain conditions.

Experimental results show DAL's efficiency in large-scale logistic regression.

DAL outperforms previous algorithms on synthetic and benchmark datasets.

Abstract

We analyze the convergence behaviour of a recently proposed algorithm for regularized estimation called Dual Augmented Lagrangian (DAL). Our analysis is based on a new interpretation of DAL as a proximal minimization algorithm. We theoretically show under some conditions that DAL converges super-linearly in a non-asymptotic and global sense. Due to a special modelling of sparse estimation problems in the context of machine learning, the assumptions we make are milder and more natural than those made in conventional analysis of augmented Lagrangian algorithms. In addition, the new interpretation enables us to generalize DAL to wide varieties of sparse estimation problems. We experimentally confirm our analysis in a large scale $\ell_1$-regularized logistic regression problem and extensively compare the efficiency of DAL algorithm to previously proposed algorithms on both synthetic and benchmark datasets.