A Randomised Subspace Gauss-Newton Method for Nonlinear Least-Squares

TL;DR

This paper introduces a randomized subspace Gauss-Newton algorithm for nonlinear least-squares problems, providing convergence guarantees and promising numerical results in logistic and nonlinear regression tasks.

Contribution

It develops a novel randomized subspace approach for Gauss-Newton, with theoretical convergence analysis and initial empirical validation.

Findings

Sublinear global convergence rate with high probability.

Effective in logistic and nonlinear regression problems.

Promising preliminary numerical results.

Abstract

We propose a Randomised Subspace Gauss-Newton (R-SGN) algorithm for solving nonlinear least-squares optimization problems, that uses a sketched Jacobian of the residual in the variable domain and solves a reduced linear least-squares on each iteration. A sublinear global rate of convergence result is presented for a trust-region variant of R-SGN, with high probability, which matches deterministic counterpart results in the order of the accuracy tolerance. Promising preliminary numerical results are presented for R-SGN on logistic regression and on nonlinear regression problems from the CUTEst collection.