Accelerated $AB$/Push-Pull Methods for Distributed Optimization over Time-Varying Directed Networks

TL;DR

This paper introduces accelerated $AB$/Push-Pull algorithms for distributed optimization over time-varying directed networks, achieving linear convergence with explicit parameter bounds and demonstrating improved performance through numerical experiments.

Contribution

It develops accelerated $AB$/Push-Pull methods with momentum techniques for time-varying directed networks, proving their linear convergence and providing explicit parameter bounds.

Findings

Achieves linear convergence for time-varying directed networks.

Provides explicit bounds for step-size and momentum parameters.

Numerical results show improved convergence with acceleration techniques.

Abstract

This paper investigates a novel approach for solving the distributed optimization problem in which multiple agents collaborate to find the global decision that minimizes the sum of their individual cost functions. First, the $AB$/Push-Pull gradient-based algorithm is considered, which employs row- and column-stochastic weights simultaneously to track the optimal decision and the gradient of the global cost function, ensuring consensus on the optimal decision. Building on this algorithm, we then develop a general algorithm that incorporates acceleration techniques, such as heavy-ball momentum and Nesterov momentum, as well as their combination with non-identical momentum parameters. Previous literature has established the effectiveness of acceleration methods for various gradient-based distributed algorithms and demonstrated linear convergence for static directed communication networks. In contrast, we focus on time-varying directed communication networks and establish linear convergence of the methods to the optimal solution, when the agents' cost functions are smooth and strongly convex. Additionally, we provide explicit bounds for the step-size value and momentum parameters, based on the properties of the cost functions, the mixing matrices, and the graph connectivity structures. Our numerical results illustrate the benefits of the proposed acceleration techniques on the $AB$/Push-Pull algorithm.