Properties of Pseudocontractive Updates in Convex Optimization

TL;DR

This paper identifies pseudocontractivity as a common property in many convex optimization methods, providing a unified analysis and convergence bounds for algorithms with such updates.

Contribution

It reveals pseudocontractivity as a unifying property across various optimization algorithms and offers a new convergence analysis framework.

Findings

Pseudocontractive updates are common in many convex optimization methods.

The property is preserved under composition and convex combination.

Error bounds show convergence rate of o(1/N) for methods with pseudocontractive updates.

Abstract

Many convex optimization methods are conceived of and analyzed in a largely separate fashion. In contrast to this traditional separation, this manuscript points out and demonstrates the utility of an important but largely unremarked common thread running through many prominent optimization methods. In particular, we show that methods such as successive orthogonal projection, gradient descent, projected gradient descent, the proximal-point method, forward-backward splitting, the alternating direction method of multipliers, and under- or over-relaxed variants of the preceding all involve updates that are of a common type --- namely, the updates satisfy a property known as pseudocontractivity. Moreover, since the property of pseudocontractivity is preserved under both composition and convex combination, updates constructed via these operations from pseudocontractive updates are themselves pseudocontractive. Having demonstrated that pseudocontractive updates are to be found in many optimization methods, we then provide a unified basic analysis of methods with pseudocontractive updates. Specifically, we prove a novel bound satisfied by the norm of the difference in iterates of pseudocontractive updates and we then use this bound to establish that the error criterion $\left\Vert x^{N}-Tx^{N}\right\Vert ^{2}$ is $o(1/N)$ for any method involving pseudocontractive updates (where $N$ is the number of iterations and $T$ is the iteration operator).