Optimization for Classical Machine Learning Problems on the GPU

TL;DR

This paper extends the GENO framework to efficiently solve constrained optimization problems on GPUs, significantly outperforming existing CPU-based and GPU-accelerated solvers in classical machine learning tasks.

Contribution

The authors develop a GPU-compatible extension of GENO that automatically generates solvers from user-friendly problem specifications, enabling fast GPU-based constrained optimization.

Findings

GPU implementation outperforms CPU-based solvers by orders of magnitude.

The framework simplifies modeling of constrained problems with an easy-to-read language.

Experimental results demonstrate substantial speedups over state-of-the-art methods.

Abstract

Constrained optimization problems arise frequently in classical machine learning. There exist frameworks addressing constrained optimization, for instance, CVXPY and GENO. However, in contrast to deep learning frameworks, GPU support is limited. Here, we extend the GENO framework to also solve constrained optimization problems on the GPU. The framework allows the user to specify constrained optimization problems in an easy-to-read modeling language. A solver is then automatically generated from this specification. When run on the GPU, the solver outperforms state-of-the-art approaches like CVXPY combined with a GPU-accelerated solver such as cuOSQP or SCS by a few orders of magnitude.