Decentralized Relaxed Smooth Optimization with Gradient Descent Methods

TL;DR

This paper introduces a novel decentralized gradient descent framework under relaxed $(L_0,L_1)$-smoothness, achieving improved convergence rates and bridging theoretical advances with practical applications in modern tasks like deep learning.

Contribution

It is the first to extend $(L_0,L_1)$-smoothness to decentralized optimization, providing new analysis techniques and adaptive methods with state-of-the-art convergence guarantees.

Findings

Achieves best-known convergence rates for convex/nonconvex functions.

Provides complexity bounds in stochastic settings with conditions for improvement.

Empirical results validate gradient-norm-dependent smoothness in real datasets.

Abstract

$L_0$-smoothness, which has been pivotal to advancing decentralized optimization theory, is often fairly restrictive for modern tasks like deep learning. The recent advent of relaxed $(L_0,L_1)$-smoothness condition enables improved convergence rates for gradient methods. Despite centralized advances, its decentralized extension remains unexplored and challenging. In this work, we propose the first general framework for decentralized gradient descent (DGD) under $(L_0,L_1)$-smoothness by introducing novel analysis techniques. For deterministic settings, our method with adaptive clipping achieves the best-known convergence rates for convex/nonconvex functions without prior knowledge of $L_0$ and $L_1$ and bounded gradient assumption. In stochastic settings, we derive complexity bounds and identify conditions for improved complexity bound in convex optimization. The empirical validation with real datasets demonstrates gradient-norm-dependent smoothness, bridging theory and practice for $(L_0,L_1)$-decentralized optimization algorithms.