Modified Step Size for Enhanced Stochastic Gradient Descent: Convergence and Experiments

TL;DR

This paper proposes a modified decay step size for stochastic gradient descent that incorporates a logarithmic term, improving convergence rates and accuracy in image classification tasks on datasets like FashionMNIST and CIFAR10.

Contribution

It introduces a novel step size schedule with a logarithmic component and provides theoretical convergence analysis for non-convex functions.

Findings

Achieves a convergence rate of O(ln T / √T) for smooth non-convex functions.

Demonstrates accuracy improvements of 0.5% on FashionMNIST.

Shows accuracy improvements of 1.4% on CIFAR10.

Abstract

This paper introduces a novel approach to enhance the performance of the stochastic gradient descent (SGD) algorithm by incorporating a modified decay step size based on $\frac{1}{\sqrt{t}}$. The proposed step size integrates a logarithmic term, leading to the selection of smaller values in the final iterations. Our analysis establishes a convergence rate of $O(\frac{\ln T}{\sqrt{T}})$ for smooth non-convex functions without the Polyak-{\L}ojasiewicz condition. To evaluate the effectiveness of our approach, we conducted numerical experiments on image classification tasks using the FashionMNIST, and CIFAR10 datasets, and the results demonstrate significant improvements in accuracy, with enhancements of $0.5\%$ and $1.4\%$ observed, respectively, compared to the traditional $\frac{1}{\sqrt{t}}$ step size. The source code can be found at \\\url{https://github.com/Shamaeem/LNSQRTStepSize}.