Understanding Limitation of Two Symmetrized Orders by Worst-case Complexity

TL;DR

This paper proves that symmetric update orders like Gaussian back substitution and symmetric Gauss-Seidel do not outperform cyclic orders in worst-case convergence speed, matching the slow convergence of cyclic methods.

Contribution

It provides a theoretical and empirical analysis showing symmetric orders cannot achieve faster convergence than cyclic orders in worst-case scenarios.

Findings

Both GBS-CD and sGS-CD can be $O(n^2)$ times slower than R-CD in worst-case.

Empirical results show GBS-ADMM and sGS-ADMM are roughly $O(n^2)$ times slower than randomized ADMM.

Symmetric orders do not improve worst-case convergence speed over cyclic orders.

Abstract

Update order is one of the major design choices of block decomposition algorithms. There are at least two classes of deterministic update orders: nonsymmetric (e.g. cyclic order) and symmetric (e.g. Gaussian back substitution or symmetric Gauss-Seidel). Recently, Coordinate Descent (CD) with cyclic order was shown to be $O(n^2)$ times slower than randomized versions in the worst-case. A natural question arises: can the symmetrized orders achieve faster convergence rates than the cyclic order, or even getting close to the randomized versions? In this paper, we give a negative answer to this question. We show that both Gaussian back substitution (GBS) and symmetric Gauss-Seidel (sGS) suffer from the same slow convergence issue as the cyclic order in the worst case. In particular, we prove that for unconstrained problems, both GBS-CD and sGS-CD can be $O(n^2)$ times slower than R-CD. Despite unconstrained problems, we also empirically study linearly constrained problems with quadratic objective: we empirically demonstrate that the convergence speed of GBS-ADMM and sGS-ADMM can be roughly $O(n^2)$ times slower than randomly permuted ADMM.