MINRES: From Negative Curvature Detection to Monotonicity Properties

TL;DR

This paper explores the properties of the MINRES method, revealing its ability to detect negative curvature and exhibit monotonicity, thus proposing it as a promising alternative to CG in nonconvex optimization algorithms.

Contribution

The paper establishes new monotonicity and negative curvature detection properties of MINRES, expanding its applicability in nonconvex optimization.

Findings

MINRES can detect negative curvature directions.

MINRES exhibits useful monotonicity properties.

MINRES may outperform CG in nonconvex optimization contexts.

Abstract

The conjugate gradient method (CG) has long been the workhorse for inner-iterations of second-order algorithms for large-scale nonconvex optimization. Prominent examples include line-search based algorithms, e.g., Newton-CG, and those based on a trust-region framework, e.g., CG-Steihaug. This is mainly thanks to CG's several favorable properties, including certain monotonicity properties and its inherent ability to detect negative curvature directions, which can arise in nonconvex optimization. This is despite the fact that the iterative method-of-choice when it comes to real symmetric but potentially indefinite matrices is arguably the celebrated minimal residual (MINRES) method. However, limited understanding of similar properties implied by MINRES in such settings has restricted its applicability within nonconvex optimization algorithms. We establish several such nontrivial properties of MINRES, including certain useful monotonicity as well as an inherent ability to detect negative curvature directions. These properties allow MINRES to be considered as a potentially superior alternative to CG for all Newton-type nonconvex optimization algorithms that employ CG as their subproblem solver.