GPU Implementation of Second-Order Linear and Nonlinear Programming Solvers

TL;DR

This paper reviews and evaluates GPU-accelerated second-order optimization solvers, focusing on interior-point methods for large sparse problems, demonstrating significant speedups over CPU implementations.

Contribution

It provides a comprehensive overview of GPU-based second-order solvers, analyzing formulations, strategies, and scalability, highlighting recent advancements and current limitations.

Findings

Speedups often exceed 10x over CPU implementations.

Effective strategies for sparse Jacobian and Hessian computations on GPUs.

Scalability demonstrated on large-scale instances.

Abstract

In recent years, GPU-accelerated optimization solvers based on second-order methods (e.g., interior-point methods) have gained momentum with the advent of mature and efficient GPU-accelerated direct sparse linear solvers, such as cuDSS. This paper provides an overview of the state of the art in GPU-based second-order solvers, focusing on pivoting-free interior-point methods for large and sparse linear and nonlinear programs. We begin by highlighting the capabilities and limitations of the currently available GPU-accelerated sparse linear solvers. Next, we discuss different formulations of the Karush-Kuhn-Tucker systems for second-order methods and evaluate their suitability for pivoting-free GPU implementations. We also discuss strategies for computing sparse Jacobians and Hessians on GPUs for nonlinear programming. Finally, we present numerical experiments demonstrating the scalability of GPU-based optimization solvers. We observe speedups often exceeding 10x compared to comparable CPU implementations on large-scale instances when solved up to medium precision. Additionally, we examine the current limitations of existing approaches.