Stochastic Steffensen method

TL;DR

This paper introduces a stochastic Steffensen method that achieves super-quadratic convergence without second derivatives, suitable for large-scale optimization, and demonstrates its effectiveness through extensive experiments.

Contribution

It proposes a novel stochastic optimization method based on Steffensen's approach, achieving high convergence orders without hyperparameter tuning and generalizing the randomized Kaczmarz method.

Findings

Outperforms existing first-order methods in experiments.

Achieves convergence order of approximately 2.414 with optimal step size.

Reduces to the randomized Kaczmarz method for quadratic objectives.

Abstract

Is it possible for a first-order method, i.e., only first derivatives allowed, to be quadratically convergent? For univariate loss functions, the answer is yes -- the Steffensen method avoids second derivatives and is still quadratically convergent like Newton method. By incorporating an optimal step size we can even push its convergence order beyond quadratic to $1+\sqrt{2} \approx 2.414$. While such high convergence orders are a pointless overkill for a deterministic algorithm, they become rewarding when the algorithm is randomized for problems of massive sizes, as randomization invariably compromises convergence speed. We will introduce two adaptive learning rates inspired by the Steffensen method, intended for use in a stochastic optimization setting and requires no hyperparameter tuning aside from batch size. Extensive experiments show that they compare favorably with several existing first-order methods. When restricted to a quadratic objective, our stochastic Steffensen methods reduce to randomized Kaczmarz method -- note that this is not true for SGD or SLBFGS -- and thus we may also view our methods as a generalization of randomized Kaczmarz to arbitrary objectives.