Comparison of Minimization Methods for Rosenbrock Functions

TL;DR

This paper compares various iterative optimization methods and step-size strategies on Rosenbrock functions, providing extensive numerical tests to evaluate their performance and trade-offs.

Contribution

It offers a comprehensive comparison of gradient descent, Newton-Raphson, and conjugate gradient methods with different step-size rules on Rosenbrock functions.

Findings

Different algorithms show distinct convergence behaviors.

Step-size selection significantly impacts optimization efficiency.

Trade-offs exist between computational cost and convergence speed.

Abstract

This paper gives an in-depth review of the most common iterative methods for unconstrained optimization using two functions that belong to a class of Rosenbrock functions as a performance test. This study covers the Steepest Gradient Descent Method, the Newton-Raphson Method, and the Fletcher-Reeves Conjugate Gradient method. In addition, four different step-size selecting methods including fixed-step-size, variable step-size, quadratic-fit, and golden section method were considered. Due to the computational nature of solving minimization problems, testing the algorithms is an essential part of this paper. Therefore, an extensive set of numerical test results is also provided to present an insightful and a comprehensive comparison of the reviewed algorithms. This study highlights the differences and the trade-offs involved in comparing these algorithms.