The Fused Kernel Library: A C++ API to Develop Highly-Efficient GPU Libraries

TL;DR

This paper introduces a C++17-based methodology and library that automatically fuses GPU functions at compile time, significantly improving performance and resource utilization without manual kernel development.

Contribution

It presents a novel approach for automatic kernel fusion in GPU libraries using high-level C++ interfaces and metaprogramming, enabling arbitrary function combinations with optimized fused kernels.

Findings

Achieves speedups from 2x to over 1000x compared to traditional libraries.

Enables automatic, on-demand kernel fusion for arbitrary GPU function sequences.

Maintains high-level programmability while maximizing GPU resource utilization.

Abstract

Existing GPU libraries often struggle to fully exploit the parallel resources and on-chip memory (SRAM) of GPUs when chaining multiple GPU functions as individual kernels. While Kernel Fusion (KF) techniques like Horizontal Fusion (HF) and Vertical Fusion (VF) can mitigate this, current library implementations often require library developers to manually create fused kernels. Hence, library users rely on limited sets of pre-compiled or template-based fused kernels. This limits the use cases that can benefit from HF and VF and increases development costs. In order to solve these issues, we present a novel methodology for building GPU libraries that enables automatic on-demand HF and VF for arbitrary combinations of GPU library functions. Our methodology defines reusable, fusionable components that users combine via high-level programming interfaces. Leveraging C++17 metaprogramming features available in compilers like nvcc, our methodology generates a single and optimized fused kernel tailored to the user's specific sequence of operations at compile time, without needing a custom compiler or manual development and pre-compilation of kernel combinations. This approach abstracts low-level GPU complexities while maximizing GPU resource utilization and keeping intermediate data in SRAM. We provide an open-source implementation demonstrating significant speedups compared to traditional libraries in various benchmarks, validating the effectiveness of this methodology for improving GPU performance in the range of 2x to more than 1000x, while preserving high-level programmability.