$\bar{G}_{mst}$:An Unbiased Stratified Statistic and a Fast Gradient Optimization Algorithm Based on It

TL;DR

This paper introduces an unbiased stratified statistic G_{mst} to improve gradient optimization by addressing gradient fluctuation effects, and proposes a fast algorithm MSSG that outperforms existing methods in deep model training.

Contribution

It presents a novel unbiased stratified statistic G_{mst} for gradient estimation and a new optimization algorithm MSSG based on it, enhancing convergence speed.

Findings

MSSG outperforms other SGD-like algorithms in experiments.

Theoretical analysis confirms fast convergence of G_{mst}.

Employing MSSG improves deep model training efficiency.

Abstract

-The fluctuation effect of gradient expectation and variance caused by parameter update between consecutive iterations is neglected or confusing by current mainstream gradient optimization algorithms. The work in this paper remedy this issue by introducing a novel unbiased stratified statistic \ $\bar{G}_{mst}$\ , a sufficient condition of fast convergence for \ $\bar{G}_{mst}$\ also is established. A novel algorithm named MSSG designed based on \ $\bar{G}_{mst}$\ outperforms other sgd-like algorithms. Theoretical conclusions and experimental evidence strongly suggest to employ MSSG when training deep model.