The G\"uler-type acceleration for proximal gradient, linearized augmented Lagrangian and linearized alternating direction method of multipliers

TL;DR

This paper introduces G"uler-type acceleration techniques for proximal gradient, augmented Lagrangian, and ADMM algorithms, improving convergence rates and efficiency in solving optimization problems relevant to machine learning and data analysis.

Contribution

It proposes a unified G"uler-type acceleration framework for multiple algorithms, enhancing convergence rates and enabling simultaneous primal-dual acceleration.

Findings

Achieves $O(1/k^2)$ convergence rate for GPGM and GLALM.

Improves partial convergence rate of GLADMM from $O(1/N^{3/2})$ to $O(1/N^2)$.

Outperforms existing methods in numerical experiments.

Abstract

In this paper, we introduce the G\"uler-type acceleration technique and utilize it to propose three acceleration algorithms: the G\"uler-type accelerated proximal gradient method (GPGM), the G\"uler-type accelerated linearized augmented Lagrangian method (GLALM) and the G\"uler-type accelerated linearized alternating direction method of multipliers (GLADMM). The key idea behind these algorithms is to fully leverage the information of negative term \bm{$-\|x^k-\hat{x}^{k-1}\|^2$} in order to design the extrapolation step. This concept of using negative terms to improve acceleration can be extended to other algorithms as well. Moreover, the proposed GLALM and GLADMM enable simultaneous acceleration of both primal and dual variables. Additionally, GPGM and GLALM achieve the same convergence rate of $O(\frac{1}{k^2})$ with some existing results. Although GLADMM achieves the same total convergence rate of $O(\frac{1}{N})$ as in existing results, the partial convergence rate is improved from $O(\frac{1}{N^{3/2}})$ to $O(\frac{1}{N^2})$. To validate the effectiveness of our algorithms, we conduct numerical experiments on various problem instances, including the $\ell_1$ regularized logistic regression, quadratic programming, and compressive sensing. The experimental results indicate that our algorithms outperform existing methods in terms of efficiency. This also demonstrates the potential of the stochastic algorithmic versions of these algorithms in application areas such as statistics, machine learning, and data mining. Finally, it is worth noting that this paper aims to introduce how G\"uler's acceleration technique can be applied to gradient-based algorithms and to provide a unified and concise framework for their construction.