On the convergence of doubly stochastic Primal-Dual Hybrid Gradient Method

TL;DR

This paper introduces a doubly stochastic primal-dual hybrid gradient method for large-scale convex-concave saddle-point problems, achieving convergence guarantees and competitive practical performance.

Contribution

It extends classical PDHG to a stochastic setting with block updates on both primal and dual variables, providing convergence analysis and a restarted variant with linear convergence.

Findings

Establishes an $ ext{O}(1/K)$ ergodic convergence rate for the method.

Proves linear convergence of the restarted variant under quadratic growth conditions.

Numerical experiments show competitive performance with existing methods.

Abstract

We study a block-structured class of convex-concave saddle-point problems in which both the primal and dual variables admit natural separable decompositions. Motivated by large-scale applications where a full update on either side can be computationally expensive, we propose a doubly stochastic primal--dual hybrid gradient method (DSPDHG) that performs randomized block updates on both primal and dual variables.The method extends classical PDHG and stochastic PDHG (SPDHG) schemes in a unified manner:it reduces to deterministic PDHG when all blocks are selected and to one-sided stochastic variants when only one side is randomized. For the general convex setting, we establish an $\mathcal{O}(1/K)$ ergodic convergence rate for the expected restricted primal--dual gap under suitable blockwise step-size conditions. We further analyze a restarted variant of DSPDHG under a quadratic growth condition in terms of the smoothed primal-dual gap. Under this regularity assumption, we prove linear convergence of the restarted outer iterates. Numerical evidence is provided to show that restarted DSPDHG with standard step sizes demonstrates competitive practical performance compared with PDHG, SPDHG, and their restarted variants.