Incremental Gauss--Newton Methods with Superlinear Convergence Rates

TL;DR

This paper introduces an Incremental Gauss--Newton method with superlinear convergence for large-scale nonlinear equations, outperforming existing linear-rate methods, and includes a mini-batch extension for even faster convergence.

Contribution

The paper presents a novel incremental Gauss--Newton method with explicit superlinear convergence, applicable to large-scale problems with a finite-sum structure, and introduces a mini-batch extension for enhanced performance.

Findings

The proposed IGN method achieves superlinear convergence rates.

Numerical experiments demonstrate the method's advantages over existing approaches.

The mini-batch extension further accelerates convergence.

Abstract

This paper addresses the challenge of solving large-scale nonlinear equations with H\"older continuous Jacobians. We introduce a novel Incremental Gauss--Newton (IGN) method within explicit superlinear convergence rate, which outperforms existing methods that only achieve linear convergence rate. In particular, we formulate our problem by the nonlinear least squares with finite-sum structure, and our method incrementally iterates with the information of one component in each round. We also provide a mini-batch extension to our IGN method that obtains an even faster superlinear convergence rate. Furthermore, we conduct numerical experiments to show the advantages of the proposed methods.