Distributed Big-Data Optimization via Block-wise Gradient Tracking

TL;DR

This paper introduces a novel distributed optimization algorithm for large-scale nonconvex problems that updates only one block of variables per iteration, reducing computation and communication burdens in multi-agent networks.

Contribution

It proposes a new block-wise gradient tracking method combining Successive Convex Approximation and a perturbed push-sum consensus protocol for nonconvex distributed optimization.

Findings

Converges asymptotically to stationary solutions.

Reduces communication overhead by updating one block at a time.

Numerical results demonstrate effectiveness and impact of block size.

Abstract

We study distributed big-data nonconvex optimization in multi-agent networks. We consider the (constrained) minimization of the sum of a smooth (possibly) nonconvex function, i.e., the agents' sum-utility, plus a convex (possibly) nonsmooth regularizer. Our interest is on big-data problems in which there is a large number of variables to optimize. If treated by means of standard distributed optimization algorithms, these large-scale problems may be intractable due to the prohibitive local computation and communication burden at each node. We propose a novel distributed solution method where, at each iteration, agents update in an uncoordinated fashion only one block of the entire decision vector. To deal with the nonconvexity of the cost function, the novel scheme hinges on Successive Convex Approximation (SCA) techniques combined with a novel block-wise perturbed push-sum consensus protocol, which is instrumental to perform local block-averaging operations and tracking of gradient averages. Asymptotic convergence to stationary solutions of the nonconvex problem is established. Finally, numerical results show the effectiveness of the proposed algorithm and highlight how the block dimension impacts on the communication overhead and practical convergence speed.