Bregman Alternating Direction Method of Multipliers

TL;DR

This paper introduces Bregman ADMM (BADMM), a generalized framework for ADMM using Bregman divergences, offering faster convergence, parallelism, and improved performance over existing methods.

Contribution

The paper develops BADMM, extending ADMM with Bregman divergences, and proves its convergence, complexity, and practical advantages including speed and parallelism.

Findings

BADMM can be faster than ADMM by up to a factor of O(n/ log(n)).

BADMM enables massive parallelism suitable for GPU implementation.

BADMM outperforms commercial solvers like Gurobi in speed.

Abstract

The mirror descent algorithm (MDA) generalizes gradient descent by using a Bregman divergence to replace squared Euclidean distance. In this paper, we similarly generalize the alternating direction method of multipliers (ADMM) to Bregman ADMM (BADMM), which allows the choice of different Bregman divergences to exploit the structure of problems. BADMM provides a unified framework for ADMM and its variants, including generalized ADMM, inexact ADMM and Bethe ADMM. We establish the global convergence and the $O(1/T)$ iteration complexity for BADMM. In some cases, BADMM can be faster than ADMM by a factor of $O(n/\log(n))$. In solving the linear program of mass transportation problem, BADMM leads to massive parallelism and can easily run on GPU. BADMM is several times faster than highly optimized commercial software Gurobi.