FINDER: Stochastic Mirroring of Noisy Quasi-Newton Search and Deep Network Training

TL;DR

FINDER is a novel stochastic optimizer that combines noise-assisted global search with Newton-like local convergence, effectively handling large-dimensional, non-convex, and non-smooth problems in deep learning and physics-informed networks.

Contribution

It introduces a derivative-free, stochastic Newton-like optimization scheme called FINDER, scalable to high dimensions and applicable to diverse large-scale problems.

Findings

FINDER outperforms Adam on benchmark functions.

FINDER demonstrates promising results on deep network training.

FINDER effectively handles physics-informed deep networks.

Abstract

Our proposal is on a new stochastic optimizer for non-convex and possibly non-smooth objective functions typically defined over large dimensional design spaces. Towards this, we have tried to bridge noise-assisted global search and faster local convergence, the latter being the characteristic feature of a Newton-like search. Our specific scheme -- acronymed FINDER (Filtering Informed Newton-like and Derivative-free Evolutionary Recursion), exploits the nonlinear stochastic filtering equations to arrive at a derivative-free update that has resemblance with the Newton search employing the inverse Hessian of the objective function. Following certain simplifications of the update to enable a linear scaling with dimension and a few other enhancements, we apply FINDER to a range of problems, starting with some IEEE benchmark objective functions to a couple of archetypal data-driven problems in deep networks to certain cases of physics-informed deep networks. The performance of the new method vis-\'a-vis the well-known Adam and a few others bears evidence to its promise and potentialities for large dimensional optimization problems of practical interest.