Generalized Random Direction Newton Algorithms for Stochastic Optimization

TL;DR

This paper introduces a family of generalized Hessian estimators for stochastic optimization using random directions, improving bias properties and convergence analysis, validated through numerical experiments.

Contribution

It proposes a novel family of Hessian estimators based on random directions with adjustable bias, and analyzes their convergence in stochastic Newton methods.

Findings

Estimators with more function measurements have lower bias.

The estimators are asymptotically unbiased.

Numerical experiments confirm theoretical convergence results.

Abstract

We present a family of generalized Hessian estimators of the objective using random direction stochastic approximation (RDSA) by utilizing only noisy function measurements. The form of each estimator and the order of the bias depend on the number of function measurements. In particular, we demonstrate that estimators with more function measurements exhibit lower-order estimation bias. We show the asymptotic unbiasedness of the estimators. We also perform asymptotic and non-asymptotic convergence analyses for stochastic Newton methods that incorporate our generalized Hessian estimators. Finally, we perform numerical experiments to validate our theoretical findings.