optHIM: Hybrid Iterative Methods for Continuous Optimization in PyTorch

TL;DR

optHIM is an open-source PyTorch library offering a variety of continuous optimization algorithms that leverage autograd for flexible line-search and trust-region methods, suitable for research and educational purposes.

Contribution

It introduces a comprehensive, extensible framework integrating multiple optimization algorithms with autograd support in PyTorch, facilitating advanced research and education.

Findings

Evaluated 11 optimization variants on diverse benchmark problems.

Demonstrated strengths and trade-offs of different methods.

Showcased optHIM's flexibility and efficiency in various scenarios.

Abstract

We introduce optHIM, an open-source library of continuous unconstrained optimization algorithms implemented in PyTorch for both CPU and GPU. By leveraging PyTorch's autograd, optHIM seamlessly integrates function, gradient, and Hessian information into flexible line-search and trust-region methods. We evaluate eleven state-of-the-art variants on benchmark problems spanning convex and non-convex landscapes. Through a suite of quantitative metrics and qualitative analyses, we demonstrate each method's strengths and trade-offs. optHIM aims to democratize advanced optimization by providing a transparent, extensible, and efficient framework for research and education.