Multilayer Lookahead: a Nested Version of Lookahead

TL;DR

This paper introduces Multilayer Lookahead, an extension of the Lookahead optimizer, demonstrating its convergence and improved generalization over SGD, with empirical results showing superior performance on image classification and GAN training tasks.

Contribution

It proposes Multilayer Lookahead, a recursive extension of Lookahead, and proves its convergence, highlighting its enhanced regularization and empirical performance.

Findings

Multilayer Lookahead converges to stationary points with O(1/√T) rate.

Multilayer Lookahead outperforms Lookahead and SGD in experiments.

The method improves generalization by amplifying implicit regularization.

Abstract

In recent years, SGD and its variants have become the standard tool to train Deep Neural Networks. In this paper, we focus on the recently proposed variant Lookahead, which improves upon SGD in a wide range of applications. Following this success, we study an extension of this algorithm, the \emph{Multilayer Lookahead} optimizer, which recursively wraps Lookahead around itself. We prove the convergence of Multilayer Lookahead with two layers to a stationary point of smooth non-convex functions with $O(\frac{1}{\sqrt{T}})$ rate. We also justify the improved generalization of both Lookahead over SGD, and of Multilayer Lookahead over Lookahead, by showing how they amplify the implicit regularization effect of SGD. We empirically verify our results and show that Multilayer Lookahead outperforms Lookahead on CIFAR-10 and CIFAR-100 classification tasks, and on GANs training on the MNIST dataset.