Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis

TL;DR

This paper provides a theoretical analysis of SGDM with dynamic schedules, demonstrating how different batch size and learning rate strategies affect convergence, and validates these findings with empirical results showing improved convergence speed.

Contribution

The paper introduces a Lyapunov-based framework to analyze SGDM convergence under practical scheduling strategies, extending existing theory and providing new insights into convergence hierarchies.

Findings

Increasing batch size guarantees convergence of expected gradient norm.

Increasing both batch size and learning rate leads to faster decay.

Dynamic scheduling outperforms fixed hyperparameters in convergence speed.

Abstract

We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function. We extend the existing theoretical framework to cover three practical scheduling strategies commonly used in deep learning: a constant batch size with a decaying learning rate, an increasing batch size with a decaying learning rate, and an increasing batch size with an increasing learning rate. Our results reveal a clear hierarchy in convergence: a constant batch size does not guarantee convergence of the expected gradient norm, whereas an increasing batch size does, and simultaneously increasing both the batch size and learning rate achieves a provably faster decay. Empirical results validate our theory, showing that dynamically scheduled SGDM significantly outperforms its fixed-hyperparameter counterpart in convergence speed. We also evaluated a warm-up schedule in experiments, which empirically outperformed all other strategies in convergence behavior.