Doubly Stochastic Primal-Dual Coordinate Method for Bilinear Saddle-Point Problem

TL;DR

This paper introduces a doubly stochastic primal-dual coordinate method for solving bilinear saddle-point problems in empirical risk minimization, demonstrating faster convergence and efficiency improvements over existing methods.

Contribution

The paper presents a novel doubly stochastic primal-dual coordinate algorithm with proven linear convergence for bilinear saddle-point problems, especially effective with structured data or costly proximal mappings.

Findings

Lower overall complexity compared to existing methods

Proven linear convergence in terms of distance and objective gap

Empirical validation on multi-task large margin nearest neighbor problem

Abstract

We propose a doubly stochastic primal-dual coordinate optimization algorithm for empirical risk minimization, which can be formulated as a bilinear saddle-point problem. In each iteration, our method randomly samples a block of coordinates of the primal and dual solutions to update. The linear convergence of our method could be established in terms of 1) the distance from the current iterate to the optimal solution and 2) the primal-dual objective gap. We show that the proposed method has a lower overall complexity than existing coordinate methods when either the data matrix has a factorized structure or the proximal mapping on each block is computationally expensive, e.g., involving an eigenvalue decomposition. The efficiency of the proposed method is confirmed by empirical studies on several real applications, such as the multi-task large margin nearest neighbor problem.