Insum: Sparse GPU Kernels Simplified and Optimized with Indirect Einsums

TL;DR

Insum introduces a new method for expressing and optimizing sparse GPU computations using indirect Einsums, resulting in significant speedups and code reduction compared to manual implementations.

Contribution

The paper presents a novel approach with format-conscious indirect Einsums and a compiler that efficiently generates GPU code, improving performance and reducing development effort.

Findings

Achieves 1.14x to 3.81x speedups on sparse GPU applications.

Reduces code complexity by up to 4491x compared to hand-written code.

Introduces fixed-length sparse formats GroupCOO and BlockGroupCOO.

Abstract

Programming high-performance sparse GPU kernels is notoriously difficult, requiring both substantial effort and deep expertise. Sparse compilers aim to simplify this process, but existing systems fall short in two key ways. First, they are primarily designed for CPUs and rarely produce high-performance GPU code. Second, when computations involve both sparse and dense regions, these compilers often fail to optimize the dense portions effectively. In this paper, we propose a new approach for expressing sparse computations. We start from format-agnostic Einsums over sparse tensors and rewrite them into format-conscious indirect Einsums, which explicitly encode format information by mapping sparse data and metadata onto dense tensor operations through indirect indexing. To execute indirect Einsums, we introduce the Insum compiler, which generates efficient GPU code for these Einsums by lowering to the PyTorch compiler, extended to better support Tensor Core-enabled indirect Einsums. We also present two fixed-length sparse formats, GroupCOO and BlockGroupCOO, designed to fit naturally with indirect Einsums. Our approach achieves 1.14x to 3.81x speedups across a range of sparse GPU applications while reducing lines of code by 202x to 4491x compared to hand-written implementations.