Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGD

TL;DR

This paper provides a theoretical analysis of hierarchical SGD, revealing how local aggregation improves convergence through a novel divergence framework, applicable to multi-level communication networks with non-IID data and non-convex objectives.

Contribution

It introduces the concepts of upward and downward divergences and analyzes the convergence of multi-level H-SGD, explaining the benefits of local aggregation in complex distributed settings.

Findings

Convergence upper bound for two-level H-SGD with non-IID data and non-convex objectives.

Identification of 'sandwich behavior' where H-SGD's convergence bounds lie between single-level SGD bounds.

Extension of analysis to multi-level H-SGD confirming the persistence of the 'sandwich behavior'.

Abstract

Hierarchical SGD (H-SGD) has emerged as a new distributed SGD algorithm for multi-level communication networks. In H-SGD, before each global aggregation, workers send their updated local models to local servers for aggregations. Despite recent research efforts, the effect of local aggregation on global convergence still lacks theoretical understanding. In this work, we first introduce a new notion of "upward" and "downward" divergences. We then use it to conduct a novel analysis to obtain a worst-case convergence upper bound for two-level H-SGD with non-IID data, non-convex objective function, and stochastic gradient. By extending this result to the case with random grouping, we observe that this convergence upper bound of H-SGD is between the upper bounds of two single-level local SGD settings, with the number of local iterations equal to the local and global update periods in H-SGD, respectively. We refer to this as the "sandwich behavior". Furthermore, we extend our analytical approach based on "upward" and "downward" divergences to study the convergence for the general case of H-SGD with more than two levels, where the "sandwich behavior" still holds. Our theoretical results provide key insights of why local aggregation can be beneficial in improving the convergence of H-SGD.