A Sub-sampled Tensor Method for Non-convex Optimization

TL;DR

This paper introduces a stochastic sub-sampled tensor method for non-convex optimization that efficiently finds approximate third-order critical points using fewer resources by leveraging a novel tensor concentration inequality.

Contribution

It proposes a new sub-sampled tensor optimization algorithm with theoretical guarantees matching deterministic methods, including a novel tensor concentration inequality.

Findings

Achieves third-order critical points in near-optimal iteration complexity.

Introduces a tensor concentration inequality for sums of tensors.

Demonstrates the method's efficiency on non-convex problems.

Abstract

We present a stochastic optimization method that uses a fourth-order regularized model to find local minima of smooth and potentially non-convex objective functions with a finite-sum structure. This algorithm uses sub-sampled derivatives instead of exact quantities. The proposed approach is shown to find an $(\epsilon_1,\epsilon_2,\epsilon_3)$-third-order critical point in at most $\bigO\left(\max\left(\epsilon_1^{-4/3}, \epsilon_2^{-2}, \epsilon_3^{-4}\right)\right)$ iterations, thereby matching the rate of deterministic approaches. In order to prove this result, we derive a novel tensor concentration inequality for sums of tensors of any order that makes explicit use of the finite-sum structure of the objective function.