SVD-Preconditioned Gradient Descent Method for Solving Nonlinear Least Squares Problems

TL;DR

This paper presents a new SVD-preconditioned gradient descent algorithm for nonlinear least squares problems, combining SVD-based preconditioning with Adam's adaptive learning rates, and demonstrates its superior convergence and accuracy in various tasks.

Contribution

The paper introduces a novel SVD-based preconditioning technique integrated with Adam, providing theoretical convergence guarantees and improved practical performance.

Findings

Faster convergence than standard Adam.

Lower error rates in regression and classification.

Effective across diverse tasks including PDE solving and image classification.

Abstract

This paper introduces a novel optimization algorithm designed for nonlinear least-squares problems. The method is derived by preconditioning the gradient descent direction using the Singular Value Decomposition (SVD) of the Jacobian. This SVD-based preconditioner is then integrated with the first- and second-moment adaptive learning rate mechanism of the Adam optimizer. We establish the local linear convergence of the proposed method under standard regularity assumptions and prove global convergence for a modified version of the algorithm under suitable conditions. The effectiveness of the approach is demonstrated experimentally across a range of tasks, including function approximation, partial differential equation (PDE) solving, and image classification on the CIFAR-10 dataset. Results show that the proposed method consistently outperforms standard Adam, achieving faster convergence and lower error in both regression and classification settings.