Nonlinear conjugate gradient methods: worst-case convergence rates via computer-assisted analyses

TL;DR

This paper introduces a computer-assisted method to analyze the worst-case convergence rates of nonlinear conjugate gradient methods, providing new bounds and insights into their performance on smooth strongly convex problems.

Contribution

It develops a novel computer-assisted framework to establish the first non-asymptotic convergence bounds for Fletcher-Reeves and improved bounds for Polak-Ribière-Polyak methods.

Findings

First non-asymptotic convergence bound for Fletcher-Reeves method

Improved non-asymptotic convergence bound for Polak-Ribière-Polyak method

Adversarial examples showing these methods can perform no better than gradient descent

Abstract

We propose a computer-assisted approach to the analysis of the worst-case convergence of nonlinear conjugate gradient methods (NCGMs). Those methods are known for their generally good empirical performances for large-scale optimization, while having relatively incomplete analyses. Using our computer-assisted approach, we establish novel complexity bounds for the Polak-Ribi\`ere-Polyak (PRP) and the Fletcher-Reeves (FR) NCGMs for smooth strongly convex minimization. In particular, we construct mathematical proofs that establish the first non-asymptotic convergence bound for FR (which is historically the first developed NCGM), and a much improved non-asymptotic convergence bound for PRP. Additionally, we provide simple adversarial examples on which these methods do not perform better than gradient descent with exact line search, leaving very little room for improvements on the same class of problems.