Distributed Big-Data Optimization via Block Communications

TL;DR

This paper introduces a novel distributed optimization algorithm for large-scale multi-agent problems that reduces communication overhead by allowing agents to optimize and communicate only parts of their variables, ensuring convergence to stationary solutions.

Contribution

It presents the first distributed method combining block-wise variable updates with success convex approximation for nonconvex problems, improving efficiency in high-dimensional settings.

Findings

Algorithm effectively reduces communication costs.

Converges to stationary solutions in nonconvex settings.

Numerical results demonstrate practical efficiency and impact of block size.

Abstract

We study distributed multi-agent large-scale optimization problems, wherein the cost function is composed of a smooth possibly nonconvex sum-utility plus a DC (Difference-of-Convex) regularizer. We consider the scenario where the dimension of the optimization variables is so large that optimizing and/or transmitting the entire set of variables could cause unaffordable computation and communication overhead. To address this issue, we propose the first distributed algorithm whereby agents optimize and communicate only a portion of their local variables. The scheme hinges on successive convex approximation (SCA) to handle the nonconvexity of the objective function, coupled with a novel block-signal tracking scheme, aiming at locally estimating the average of the agents' gradients. Asymptotic convergence to stationary solutions of the nonconvex problem is established. Numerical results on a sparse regression problem show the effectiveness of the proposed algorithm and the impact of the block size on its practical convergence speed and communication cost.