Differentiation of inertial methods for optimizing smooth parametric function

TL;DR

This paper analyzes inertial optimization methods for smooth, strongly convex functions depending on a parameter, focusing on their convergence, derivative stability, and the rate of convergence of the derivative with respect to the parameter.

Contribution

It introduces a unified analysis of inertial methods, proving convergence, derivative stability, and local linear convergence rates for the parameter-dependent optimization problem.

Findings

Sequences converge to the unique minimizer.

Derivative of the sequence converges to the derivative of the limit.

Convergence rate is locally linear with diminishing error.

Abstract

In this paper, we consider the minimization of a $C^2-$smooth and strongly convex objective depending on a given parameter, which is usually found in many practical applications. We suppose that we desire to solve the problem with some inertial methods which cover a broader existing well-known inertial methods. Our main goal is to analyze the derivative of this algorithm as an infinite iterative process in the sense of ``automatic'' differentiation. This procedure is very common and has gain more attention recently. From a pure optimization perspective and under some mild premises, we show that any sequence generated by these inertial methods converge to the unique minimizer of the problem, which depends on the parameter. Moreover, we show a local linear convergence rate of the generated sequence. Concerning the differentiation of the scheme, we prove that the derivative of the sequence with respect to the parameter converges to the derivative of the limit of the sequence showing that any sequence is <<derivative stable>>. Finally, we investigate the rate at which the convergence occurs. We show that, this is locally linear with an error term tending to zero.