A stochastic first-order method with multi-extrapolated momentum for highly smooth unconstrained optimization

TL;DR

This paper introduces a novel stochastic first-order method with multi-extrapolated momentum that leverages high-order smoothness of the objective function to accelerate unconstrained optimization, achieving improved sample complexity.

Contribution

It proposes the first SFOM that exploits arbitrary-order smoothness for acceleration, with theoretical guarantees and practical validation.

Findings

Achieves a sample complexity of d( p+1)/p) for e9psilon-approximate stationary points.

Demonstrates acceleration over existing methods by exploiting high-order smoothness.

Preliminary experiments confirm the practical effectiveness of the proposed method.

Abstract

In this paper, we consider an unconstrained stochastic optimization problem where the objective function exhibits high-order smoothness. Specifically, we propose a new stochastic first-order method (SFOM) with multi-extrapolated momentum, in which multiple extrapolations are performed in each iteration, followed by a momentum update based on these extrapolations. We demonstrate that the proposed SFOM can accelerate optimization by exploiting the high-order smoothness of the objective function $f$. Assuming that the $p$th-order derivative of $f$ is Lipschitz continuous for some $p\ge2$, and under additional mild assumptions, we establish that our method achieves a sample complexity of $\widetilde{\mathcal{O}}(\epsilon^{-(3p+1)/p})$ for finding a point $x$ such that $\mathbb{E}[\|\nabla f(x)\|]\le\epsilon$. To the best of our knowledge, this is the first SFOM to leverage arbitrary-order smoothness of the objective function for acceleration, resulting in a sample complexity that improves upon the best-known results without assuming the mean-squared smoothness condition. Preliminary numerical experiments validate the practical performance of our method and support our theoretical findings.