dCSR: A Memory-Efficient Sparse Matrix Representation for Parallel Neural Network Inference

TL;DR

This paper introduces dCSR, a memory-efficient sparse matrix format optimized for embedded neural network inference on SIMD-capable microcontrollers, achieving significant throughput improvements.

Contribution

We develop Delta-Compressed Storage Row (dCSR), a novel sparse matrix representation that reduces memory overhead and accelerates inference on embedded SIMD hardware.

Findings

Achieves up to 2.9x throughput increase for SpMV

Provides competitive compression ratios

Demonstrates effectiveness on ARM Cortex-M55 MCU

Abstract

Reducing the memory footprint of neural networks is a crucial prerequisite for deploying them in small and low-cost embedded devices. Network parameters can often be reduced significantly through pruning. We discuss how to best represent the indexing overhead of sparse networks for the coming generation of Single Instruction, Multiple Data (SIMD)-capable microcontrollers. From this, we develop Delta-Compressed Storage Row (dCSR), a storage format for sparse matrices that allows for both low overhead storage and fast inference on embedded systems with wide SIMD units. We demonstrate our method on an ARM Cortex-M55 MCU prototype with M-Profile Vector Extension(MVE). A comparison of memory consumption and throughput shows that our method achieves competitive compression ratios and increases throughput over dense methods by up to $2.9 \times$ for sparse matrix-vector multiplication (SpMV)-based kernels and $1.06 \times$ for sparse matrix-matrix multiplication (SpMM). This is accomplished through handling the generation of index information directly in the SIMD unit, leading to an increase in effective memory bandwidth.