Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities

TL;DR

This paper introduces a novel combination of similarity, compression, and local steps techniques to improve communication efficiency in distributed variational inequality problems, applicable to large-scale machine learning tasks.

Contribution

It presents the first integrated approach combining three techniques for variational inequalities, achieving superior theoretical communication complexity guarantees.

Findings

The proposed methods outperform existing approaches in communication efficiency.

Theoretical guarantees are validated through experiments on synthetic and real datasets.

Significant reduction in communication rounds and costs in distributed settings.

Abstract

Variational inequalities are a broad and flexible class of problems that includes minimization, saddle point, and fixed point problems as special cases. Therefore, variational inequalities are used in various applications ranging from equilibrium search to adversarial learning. With the increasing size of data and models, today's instances demand parallel and distributed computing for real-world machine learning problems, most of which can be represented as variational inequalities. Meanwhile, most distributed approaches have a significant bottleneck - the cost of communications. The three main techniques to reduce the total number of communication rounds and the cost of one such round are the similarity of local functions, compression of transmitted information, and local updates. In this paper, we combine all these approaches. Such a triple synergy did not exist before for variational inequalities and saddle problems, nor even for minimization problems. The methods presented in this paper have the best theoretical guarantees of communication complexity and are significantly ahead of other methods for distributed variational inequalities. The theoretical results are confirmed by adversarial learning experiments on synthetic and real datasets.