Adjusted Shuffling SARAH: Advancing Complexity Analysis via Dynamic Gradient Weighting

TL;DR

This paper introduces Adjusted Shuffling SARAH, a new variance-reduction algorithm with dynamic gradient weighting that improves complexity bounds in convex optimization, and an inexact variant reducing computational costs.

Contribution

It presents a novel shuffling-based variance reduction method with dynamic weighting and analyzes its complexity, also proposing an inexact version for large-scale problems.

Findings

Achieves the best-known gradient complexity for shuffling variance reduction methods.

Applicable to any shuffling technique, narrowing the complexity gap.

Inexact variant retains linear convergence with reduced computational cost.

Abstract

In this paper, we propose Adjusted Shuffling SARAH, a novel algorithm that integrates shuffling techniques with the well-known variance-reduced algorithm SARAH while dynamically adjusting the stochastic gradient weights in each update to enhance exploration. Our method achieves the best-known gradient complexity for shuffling variance reduction methods in a strongly convex setting. This result applies to any shuffling technique, which narrows the gap in the complexity analysis of variance reduction methods between uniform sampling and shuffling data. Furthermore, we introduce Inexact Adjusted Reshuffling SARAH, an inexact variant of Adjusted Shuffling SARAH that eliminates the need for full-batch gradient computations. This algorithm retains the same linear convergence rate as Adjusted Shuffling SARAH while showing an advantage in total complexity when the sample size is very large.