On the insufficiency of existing momentum schemes for Stochastic Optimization

TL;DR

This paper demonstrates that popular momentum-based stochastic gradient methods like HB and NAG can fail to outperform SGD on simple problems, and introduces ASGD, a new algorithm that empirically outperforms existing methods.

Contribution

The paper proves the limitations of HB and NAG in stochastic settings and introduces ASGD, a simple alternative that shows improved performance.

Findings

HB and NAG do not always outperform SGD on simple instances.

ASGD significantly outperforms HB, NAG, and SGD in empirical tests.

Momentum methods' gains are mainly due to mini-batching, not inherent advantages.

Abstract

Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide significant improvements over stochastic gradient descent (SGD). Rigorously speaking, "fast gradient" methods have provable improvements over gradient descent only for the deterministic case, where the gradients are exact. In the stochastic case, the popular explanations for their wide applicability is that when these fast gradient methods are applied in the stochastic case, they partially mimic their exact gradient counterparts, resulting in some practical gain. This work provides a counterpoint to this belief by proving that there exist simple problem instances where these methods cannot outperform SGD despite the best setting of its parameters. These negative problem instances are, in an informal sense, generic; they do not look like carefully constructed pathological instances. These results suggest (along with empirical evidence) that HB or NAG's practical performance gains are a by-product of mini-batching. Furthermore, this work provides a viable (and provable) alternative, which, on the same set of problem instances, significantly improves over HB, NAG, and SGD's performance. This algorithm, referred to as Accelerated Stochastic Gradient Descent (ASGD), is a simple to implement stochastic algorithm, based on a relatively less popular variant of Nesterov's Acceleration. Extensive empirical results in this paper show that ASGD has performance gains over HB, NAG, and SGD.