Convergence Rate Analysis for Fixed-Point Iterations of Generalized Averaged Nonexpansive Operators

TL;DR

This paper introduces the generalized averaged nonexpansive (GAN) operator, analyzes its convergence rates in fixed-point iterations, and applies the results to convex optimization problems common in data science.

Contribution

It defines the GAN operator with a positive exponent, provides convergence rate analysis, and applies the theory to optimize convex problems in data science.

Findings

GAN operators with positive exponents converge to fixed points.

Fixed-point iterations of GAN operators with exponents less than 1 achieve exponential convergence.

Global convergence rates depend on the exponents and Hölder regularity.

Abstract

We estimate convergence rates for fixed-point iterations of a class of nonlinear operators which are partially motivated from solving convex optimization problems. We introduce the notion of the generalized averaged nonexpansive (GAN) operator with a positive exponent, and provide a convergence rate analysis of the fixed-point iteration of the GAN operator. The proposed generalized averaged nonexpansiveness is weaker than the averaged nonexpansiveness while stronger than nonexpansiveness. We show that the fixed-point iteration of a GAN operator with a positive exponent converges to its fixed-point and estimate the local convergence rate (the convergence rate in terms of the distance between consecutive iterates) according to the range of the exponent. We prove that the fixed-point iteration of a GAN operator with a positive exponent strictly smaller than 1 can achieve an exponential global convergence rate (the convergence rate in terms of the distance between an iterate and the solution). Furthermore, we establish the global convergence rate of the fixed-point iteration of a GAN operator, depending on both the exponent of generalized averaged nonexpansiveness and the exponent of the H$\ddot{\text{o}}$lder regularity, if the GAN operator is also H$\ddot{\text{o}}$lder regular. We then apply the established theory to three types of convex optimization problems that appear often in data science to design fixed-point iterative algorithms for solving these optimization problems and to analyze their convergence properties.