A more direct and better variant of New Q-Newton's method Backtracking for m equations in m variables

TL;DR

This paper introduces improved variants of New Q-Newton's method for solving systems of equations, demonstrating better saddle point avoidance and convergence properties through modifications of Levenberg-Marquardt and Q-Newton's methods.

Contribution

It proposes new algorithms based on New Q-Newton's method with enhanced saddle point avoidance and convergence guarantees for systems of equations.

Findings

New algorithms outperform existing methods in avoiding saddle points.

Proved global convergence and saddle point avoidance for the modified Levenberg-Marquardt.

Enhanced method better suited for solving systems of equations with improved guarantees.

Abstract

In this paper we apply the ideas of New Q-Newton's method directly to a system of equations, utilising the specialties of the cost function $f=||F||^2$, where $F=(f_1,\ldots ,f_m)$. The first algorithm proposed here is a modification of Levenberg-Marquardt algorithm, where we prove some new results on global convergence and avoidance of saddle points. The second algorithm proposed here is a modification of New Q-Newton's method Backtracking, where we use the operator $\nabla ^2f(x)+\delta ||F(x)||^{\tau}$ instead of $\nabla ^2f(x)+\delta ||\nabla f(x)||^{\tau}$. This new version is more suitable than New Q-Newton's method Backtracking itself, while currently has better avoidance of saddle points guarantee than Levenberg-Marquardt algorithms. Also, a general scheme for second order methods for solving systems of equations is proposed. We will also discuss a way to avoid that the limit of the constructed sequence is a solution of $H(x)^{\intercal}F(x)=0$ but not of $F(x)=0$.