Decentralized Learning with Dynamically Refined Edge Weights: A Data-Dependent Framework

TL;DR

This paper introduces a data-dependent, decentralized optimization framework called D3GD that dynamically refines edge weights based on data similarity, significantly accelerating convergence in directed graph settings.

Contribution

The paper proposes a novel data-dependent edge weight refinement strategy for decentralized learning, improving convergence speed over classical methods.

Findings

D3GD accelerates convergence by 30-40% compared to Di-DGD.

The framework learns adaptive edge weights based on data similarity.

Numerical experiments validate the effectiveness of the proposed method.

Abstract

This paper aims to accelerate decentralized optimization by strategically designing the edge weights used in the agent-to-agent message exchanges. We propose a Dynamic Directed Decentralized Gradient (D3GD) framework and show that the proposed data-dependent framework is a practical alternative to the classical directed DGD (Di-DGD) algorithm for learning on directed graphs. To obtain a strategy for edge weights refinement, we derive a design function inspired by the cost-to-go function in a new convergence analysis for Di-DGD. This results in a data-dependent dynamical design for the edge weights. A fully decentralized version of D3GD is developed such that each agent refines its communication strategy using only neighbor's information. Numerical experiments show that D3GD accelerates convergence towards stationary solution by 30-40\% over Di-DGD, and learns edge weights that adapt to data similarity.