Numerical Methods for Distributed Stochastic Compositional Optimization Problems with Aggregative Structure

TL;DR

This paper introduces a novel distributed stochastic compositional gradient descent method leveraging hybrid variance reduction and dynamic consensus, achieving optimal convergence rates for aggregative problems, with applications demonstrated in reinforcement learning.

Contribution

It proposes a new distributed aggregative stochastic compositional gradient descent method with optimal convergence rates, incorporating communication compression for efficiency.

Findings

Achieves optimal convergence rate of O(K^{-1/2}) for smooth functions.

Effectively tracks private expectation functions using hybrid variance reduction.

Validated through decentralized reinforcement learning experiments.

Abstract

The paper studies the distributed stochastic compositional optimization problems over networks, where all the agents' inner-level function is the sum of each agent's private expectation function. Focusing on the aggregative structure of the inner-level function, we employ the hybrid variance reduction method to obtain the information on each agent's private expectation function, and apply the dynamic consensus mechanism to track the information on each agent's inner-level function. Then by combining with the standard distributed stochastic gradient descent method, we propose a distributed aggregative stochastic compositional gradient descent method. When the objective function is smooth, the proposed method achieves the optimal convergence rate $\mathcal{O}\left(K^{-1/2}\right)$. We further combine the proposed method with the communication compression and propose the communication compressed variant distributed aggregative stochastic compositional gradient descent method. The compressed variant of the proposed method maintains the optimal convergence rate $\mathcal{O}\left(K^{-1/2}\right)$. Simulated experiments on decentralized reinforcement learning verify the effectiveness of the proposed methods.