A Unified Analysis of Stochastic Gradient Descent with Arbitrary Data Permutations and Beyond

TL;DR

This paper develops a unified convergence analysis framework for permutation-based SGD algorithms, encompassing various permutation types and extending to federated learning scenarios with arbitrary client permutations.

Contribution

It introduces a general assumption capturing inter-epoch permutation dependencies, unifying analysis across all permutation categories including dependent permutations.

Findings

Unified framework for permutation-based SGD algorithms.

Extension to federated learning with arbitrary client permutations.

Addresses inter-epoch permutation dependencies in analysis.

Abstract

We aim to provide a unified convergence analysis for permutation-based Stochastic Gradient Descent (SGD), where data examples are permuted before each epoch. By examining the relations among permutations, we categorize existing permutation-based SGD algorithms into four categories: Arbitrary Permutations, Independent Permutations (including Random Reshuffling), One Permutation (including Incremental Gradient, Shuffle One and Nice Permutation) and Dependent Permutations (including GraBs Lu et al., 2022; Cooper et al., 2023). Existing unified analyses failed to encompass the Dependent Permutations category due to the inter-epoch dependencies in its permutations. In this work, we propose a general assumption that captures the inter-epoch permutation dependencies. Using the general assumption, we develop a unified framework for permutation-based SGD with arbitrary permutations of examples, incorporating all the aforementioned representative algorithms. Furthermore, we adapt our framework on example ordering in SGD for client ordering in Federated Learning (FL). Specifically, we develop a unified framework for regularized-participation FL with arbitrary permutations of clients.