Residual subspace evolution strategies for nonlinear inverse problems

TL;DR

This paper introduces Residual Subspace Evolution Strategies (RSES), a derivative-free optimization method for nonlinear inverse problems that efficiently builds residual-only surrogates to guide updates, outperforming traditional methods especially in non-smooth or noisy settings.

Contribution

The paper presents RSES, a novel derivative-free solver that constructs residual-based surrogates without Jacobians, enabling efficient and robust optimization in challenging inverse problems.

Findings

RSES reduces misfit effectively across various tasks.

It matches or exceeds performance of established algorithms like xNES, NEWUOA, Adam, and ensemble Kalman inversion.

The method is especially advantageous when smoothness or covariance assumptions are invalid.

Abstract

Nonlinear inverse problems pervade engineering and science, yet noisy, non-differentiable, or expensive residual evaluations routinely defeat Jacobian-based solvers. Derivative-free alternatives either demand smoothness, require large populations to stabilise covariance estimates, or stall on flat regions where gradient information fades. This paper introduces residual subspace evolution strategies (RSES), a derivative-free solver that draws Gaussian probes around the current iterate, records how residuals change along those directions, and recombines the probes through a least-squares solve to produce an optimal update. The method builds a residual-only surrogate without forming Jacobians or empirical covariances, and each iteration costs just $k+1$ residual evaluations with $O(k^3)$ linear algebra overhead, where $k$ remains far smaller than the parameter dimension. Benchmarks on calibration, regression, and deconvolution tasks show that RSES reduces misfit consistently across deterministic and stochastic settings, matching or exceeding xNES, NEWUOA, Adam, and ensemble Kalman inversion under matched evaluation budgets. The gains are most pronounced when smoothness or covariance assumptions break, suggesting that lightweight residual-difference surrogates can reliably guide descent where heavier machinery struggles.