SVGD: A Virtual Gradients Descent Method for Stochastic Optimization

TL;DR

SVGD is a novel stochastic optimization algorithm inspired by dynamic programming, utilizing automatic differentiation to efficiently compute virtual gradients, with proven convergence and demonstrated advantages over existing methods.

Contribution

Introduces SVGD, a new stochastic gradient descent method leveraging computational graphs and automatic differentiation for improved efficiency and convergence analysis.

Findings

SVGD outperforms other stochastic optimization methods on multiple datasets.

SVGD is computationally efficient with low memory requirements.

Theoretical analysis confirms convergence properties of SVGD.

Abstract

Inspired by dynamic programming, we propose Stochastic Virtual Gradient Descent (SVGD) algorithm where the Virtual Gradient is defined by computational graph and automatic differentiation. The method is computationally efficient and has little memory requirements. We also analyze the theoretical convergence properties and implementation of the algorithm. Experimental results on multiple datasets and network models show that SVGD has advantages over other stochastic optimization methods.