Hybrid Acceleration Scheme for Variance Reduced Stochastic Optimization Algorithms

TL;DR

This paper introduces a hybrid acceleration scheme that combines fast local convergence methods with globally convergent variance reduced stochastic algorithms, significantly improving convergence speed in optimization tasks.

Contribution

The paper proposes a novel hybrid acceleration scheme that integrates quasi-Newton methods with variance reduced stochastic algorithms, ensuring global convergence and faster local convergence.

Findings

Convergence of the hybrid scheme is proven almost surely.

Numerical experiments show significantly improved convergence.

The method effectively combines local and global convergence properties.

Abstract

Stochastic variance reduced optimization methods are known to be globally convergent while they suffer from slow local convergence, especially when moderate or high accuracy is needed. To alleviate this problem, we propose an optimization algorithm -- which we refer to as a hybrid acceleration scheme -- for a class of proximal variance reduced stochastic optimization algorithms. The proposed optimization scheme combines a fast locally convergent algorithm, such as a quasi--Newton method, with a globally convergent variance reduced stochastic algorithm, for instance SAGA or L--SVRG. Our global convergence result of the hybrid acceleration method is based on specific safeguard conditions that need to be satisfied for a step of the locally fast convergent method to be accepted. We prove that the sequence of the iterates generated by the hybrid acceleration scheme converges almost surely to a solution of the underlying optimization problem. We also provide numerical experiments that show significantly improved convergence of the hybrid acceleration scheme compared to the basic stochastic variance reduced optimization algorithm.