VEGETA: Vertically-Integrated Extensions for Sparse/Dense GEMM Tile Acceleration on CPUs

TL;DR

VEGETA introduces ISA and microarchitecture extensions for CPUs to efficiently support structured sparsity in deep learning workloads, significantly improving GEMM performance across various sparsity levels.

Contribution

It proposes novel extensions enabling flexible structured sparsity support in CPU matrix engines, enhancing performance for diverse deep learning models.

Findings

VEGETA achieves up to 3.74x speed-up on sparse DNN layers.

Supports diverse sparsity patterns with programmable extensions.

Outperforms state-of-the-art dense matrix engines in CPU performance.

Abstract

Deep Learning (DL) acceleration support in CPUs has recently gained a lot of traction, with several companies (Arm, Intel, IBM) announcing products with specialized matrix engines accessible via GEMM instructions. CPUs are pervasive and need to handle diverse requirements across DL workloads running in edge/HPC/cloud platforms. Therefore, as DL workloads embrace sparsity to reduce the computations and memory size of models, it is also imperative for CPUs to add support for sparsity to avoid under-utilization of the dense matrix engine and inefficient usage of the caches and registers. This work presents VEGETA, a set of ISA and microarchitecture extensions over dense matrix engines to support flexible structured sparsity for CPUs, enabling programmable support for diverse DL models with varying degrees of sparsity. Compared to the state-of-the-art (SOTA) dense matrix engine in CPUs, a VEGETA engine provides 1.09x, 2.20x, 3.74x, and 3.28x speed-ups when running 4:4 (dense), 2:4, 1:4, and unstructured (95%) sparse DNN layers.