A Stochastic Newton-type Method for Non-smooth Optimization

TL;DR

This paper develops a stochastic Quasi-Newton framework for non-smooth optimization, providing probabilistic bounds on iteration complexity without requiring unbiased Hessian estimators, with applications in tomography and image denoising.

Contribution

It introduces a novel stochastic analysis for Quasi-Newton methods applicable to non-smooth problems, relaxing traditional unbiasedness and variance assumptions.

Findings

Derived expectation and probability bounds for iteration counts

Applied the framework to tomography and image denoising tasks

Demonstrated effectiveness without requiring unbiased Hessian estimates

Abstract

We introduce a new framework for analyzing (Quasi-}Newton type methods applied to non-smooth optimization problems. The source of randomness comes from the evaluation of the (approximation) of the Hessian. We derive, using a variant of Chernoff bounds for stopping times, expectation and probability bounds for the random variable representing the number of iterations of the algorithm until approximate first order optimality conditions are validated. As an important distinction to previous results in the literature, we do not require that the estimator is unbiased or that it has finite variance. We then showcase our theoretical results in a stochastic Quasi-Newton method for X-ray free electron laser orbital tomography and in a sketched Newton method for image denoising.