A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information

TL;DR

This paper enhances the stochastic variance reduced gradient (SVRG) method by integrating Barzilai-Borwein curvature information, leading to improved convergence and performance on benchmark datasets.

Contribution

It introduces a novel SVRG variant that incorporates Barzilai-Borwein techniques for variance reduction and second-order information, with proven linear convergence.

Findings

Proposed method outperforms existing variance reduced methods on some benchmark problems.

The method demonstrates linear convergence under certain conditions.

Numerical experiments validate the efficiency of the approach.

Abstract

In this paper, we consider to improve the stochastic variance reduce gradient (SVRG) method via incorporating the curvature information of the objective function. We propose to reduce the variance of stochastic gradients using the computationally efficient Barzilai-Borwein (BB) method by incorporating it into the SVRG. We also incorporate a BB-step size as its variant. We prove its linear convergence theorem that works not only for the proposed method but also for the other existing variants of SVRG with second-order information. We conduct the numerical experiments on the benchmark datasets and show that the proposed method with constant step size performs better than the existing variance reduced methods for some test problems.