A New Kernel Regularity Condition for Distributed Mirror Descent: Broader Coverage and Simpler Analysis

TL;DR

This paper introduces a new regularity condition called Hessian relative uniform continuity (HRUC) that broadens the applicability of convergence analysis for distributed mirror descent in non-Euclidean geometries, addressing practical kernel limitations.

Contribution

The paper develops HRUC, a mild regularity condition that encompasses most standard kernels, enabling convergence guarantees for distributed mirror descent without restrictive assumptions.

Findings

HRUC is satisfied by nearly all standard kernels.

Convergence guarantees are established under HRUC for mirror descent.

Analysis extends to other decentralized optimization methods.

Abstract

Existing convergence of distributed optimization methods in non-Euclidean geometries typically rely on kernel assumptions: (i) global Lipschitz smoothness and (ii) bi-convexity of the associated Bregman divergence function. Unfortunately, these conditions are violated by nearly all kernels used in practice, leaving a huge theory-practice gap. This work closes this gap by developing a unified analytical tool that guarantees convergence under mild conditions. Specifically, we introduce Hessian relative uniform continuity (HRUC), a regularity satisfied by nearly all standard kernels. Importantly, HRUC is closed under concatenation, positive scaling, composition, and various kernel combinations. Leveraging the geometric structure induced by HRUC, we derive convergence guarantees for mirror descent-based gradient tracking without imposing any restrictive assumptions. More broadly, our analysis techniques extend seamlessly to other decentralized optimization methods in genuinely non-Euclidean and non-Lipschitz settings.