An efficient and robust SAV based algorithm for discrete gradient systems arising from optimizations

TL;DR

This paper introduces a new SAV-based minimization algorithm that is unconditionally energy diminishing, allows large step-sizes, and converges faster than traditional gradient methods, with proven convergence and practical efficiency.

Contribution

The paper presents a novel SAV-based algorithm with relaxation and adaptive strategies that improves convergence speed and robustness over existing gradient methods.

Findings

The new algorithm is unconditionally energy diminishing without parameter tuning.

It allows large step-sizes, accelerating convergence.

Numerical results show it outperforms popular gradient methods in robustness and speed.

Abstract

We propose in this paper a new minimization algorithm based on a slightly modified version of the scalar auxiliary variable (SAV) approach coupled with a relaxation step and an adaptive strategy. It enjoys several distinct advantages over popular gradient based methods: (i) it is unconditionally energy diminishing with a modified energy which is intrinsically related to the original energy, thus no parameter tuning is needed for stability; (ii) it allows the use of large step-sizes which can effectively accelerate the convergence rate. We also present a convergence analysis for some SAV based algorithms, which include our new algorithm without the relaxation step as a special case. We apply our new algorithm to several illustrative and benchmark problems, and compare its performance with several popular gradient based methods. The numerical results indicate that the new algorithm is very robust, and its adaptive version usually converges significantly faster than those popular gradient descent based methods.