Tailors: Accelerating Sparse Tensor Algebra by Overbooking Buffer Capacity

TL;DR

This paper introduces a speculative tiling method called overbooking for sparse tensor algebra accelerators, significantly improving buffer utilization and performance by allowing controlled buffer overflows with a hardware mechanism, demonstrated across diverse workloads.

Contribution

It proposes a novel overbooking approach with a hardware mechanism, Tailors, and a statistical tile size selection method, Swiftiles, to enhance buffer utilization and performance in sparse tensor accelerators.

Findings

Average speedup of 52.7x over ExTensor without tiling.

Average energy reduction of 22.5x over ExTensor.

Effective buffer utilization across 22 workloads.

Abstract

Sparse tensor algebra is a challenging class of workloads to accelerate due to low arithmetic intensity and varying sparsity patterns. Prior sparse tensor algebra accelerators have explored tiling sparse data to increase exploitable data reuse and improve throughput, but typically allocate tile size in a given buffer for the worst-case data occupancy. This severely limits the utilization of available memory resources and reduces data reuse. Other accelerators employ complex tiling during preprocessing or at runtime to determine the exact tile size based on its occupancy. This paper proposes a speculative tensor tiling approach, called overbooking, to improve buffer utilization by taking advantage of the distribution of nonzero elements in sparse tensors to construct larger tiles with greater data reuse. To ensure correctness, we propose a low-overhead hardware mechanism, Tailors, that can tolerate data overflow by design while ensuring reasonable data reuse. We demonstrate that Tailors can be easily integrated into the memory hierarchy of an existing sparse tensor algebra accelerator. To ensure high buffer utilization with minimal tiling overhead, we introduce a statistical approach, Swiftiles, to pick a tile size so that tiles usually fit within the buffer's capacity, but can potentially overflow, i.e., it overbooks the buffers. Across a suite of 22 sparse tensor algebra workloads, we show that our proposed overbooking strategy introduces an average speedup of $52.7\times$ and $2.3\times$ and an average energy reduction of $22.5\times$ and $2.5\times$ over ExTensor without and with optimized tiling, respectively.