Achieving Linear Speedup in Decentralized Stochastic Compositional Minimax Optimization

TL;DR

This paper introduces a novel decentralized stochastic compositional gradient descent ascent algorithm with momentum, achieving linear speedup in decentralized minimax problems, and demonstrates its effectiveness on imbalanced classification tasks.

Contribution

The paper develops a new algorithm that reduces consensus error in decentralized compositional minimax optimization, enabling linear speedup with multiple workers.

Findings

Achieves linear speedup in decentralized settings

Reduces consensus error with a momentum-based approach

Effective on imbalanced classification problems

Abstract

The stochastic compositional minimax problem has attracted a surge of attention in recent years since it covers many emerging machine learning models. Meanwhile, due to the emergence of distributed data, optimizing this kind of problem under the decentralized setting becomes badly needed. However, the compositional structure in the loss function brings unique challenges to designing efficient decentralized optimization algorithms. In particular, our study shows that the standard gossip communication strategy cannot achieve linear speedup for decentralized compositional minimax problems due to the large consensus error about the inner-level function. To address this issue, we developed a novel decentralized stochastic compositional gradient descent ascent with momentum algorithm to reduce the consensus error in the inner-level function. As such, our theoretical results demonstrate that it is able to achieve linear speedup with respect to the number of workers. We believe this novel algorithmic design could benefit the development of decentralized compositional optimization. Finally, we applied our methods to the imbalanced classification problem. The extensive experimental results provide evidence for the effectiveness of our algorithm.