DISH: A Distributed Hybrid Optimization Method Leveraging System Heterogeneity

TL;DR

DISH is a novel distributed optimization method that exploits system heterogeneity by allowing agents with different computational capabilities to perform tailored updates, achieving improved convergence rates.

Contribution

The paper introduces DISH, a hybrid distributed optimization method that leverages agent heterogeneity and generalizes existing methods, with theoretical convergence guarantees.

Findings

DISH achieves linear convergence in strongly-convex settings.

GRAND and Alt-GRAND algorithms have proven global sublinear and linear convergence rates.

Numerical experiments confirm the effectiveness of the proposed methods.

Abstract

We study distributed optimization problems over multi-agent networks, including consensus and network flow problems. Existing distributed methods neglect the heterogeneity among agents' computational capabilities, limiting their effectiveness. To address this, we propose DISH, a distributed hybrid method that leverages system heterogeneity. DISH allows agents with higher computational capabilities or lower computational costs to perform local Newton-type updates while others adopt simpler gradient-type updates. Notably, DISH covers existing methods like EXTRA, DIGing, and ESOM-0 as special cases. To analyze DISH's performance with general update directions, we formulate distributed problems as minimax problems and introduce GRAND (gradient-related ascent and descent) and its alternating version, Alt-GRAND, for solving these problems. GRAND generalizes DISH to centralized minimax settings, accommodating various descent ascent update directions, including gradient-type, Newton-type, scaled gradient, and other general directions, within acute angles to the partial gradients. Theoretical analysis establishes global sublinear and linear convergence rates for GRAND and Alt-GRAND in strongly-convex-nonconcave and strongly-convex-PL settings, providing linear rates for DISH. In addition, we derive the local superlinear convergence of Newton-based variations of GRAND in centralized settings. Numerical experiments validate the effectiveness of our methods.