Analyzing and Improving Greedy 2-Coordinate Updates for Equality-Constrained Optimization via Steepest Descent in the 1-Norm

TL;DR

This paper introduces an improved greedy 2-coordinate update method for equality-constrained optimization that converges faster than random selection and efficiently handles bound constraints, with applications to SVM dual problems.

Contribution

It establishes a new convergence rate for greedy coordinate updates based on the proximal Polyak-Lojasiewicz condition and develops an efficient $O(n \, \log n)$ algorithm for constrained steepest descent.

Findings

Faster convergence rate than random selection under certain conditions

Efficient $O(n \log n)$ algorithm for bound- and summation-constrained steepest descent

Improved progress per iteration over previous greedy rules

Abstract

We consider minimizing a smooth function subject to a summation constraint over its variables. By exploiting a connection between the greedy 2-coordinate update for this problem and equality-constrained steepest descent in the 1-norm, we give a convergence rate for greedy selection under a proximal Polyak-Lojasiewicz assumption that is faster than random selection and independent of the problem dimension $n$. We then consider minimizing with both a summation constraint and bound constraints, as arises in the support vector machine dual problem. Existing greedy rules for this setting either guarantee trivial progress only or require $O(n^2)$ time to compute. We show that bound- and summation-constrained steepest descent in the L1-norm guarantees more progress per iteration than previous rules and can be computed in only $O(n \log n)$ time.