A Simple Sparse Matrix Vector Multiplication Approach to Padded Convolution

TL;DR

This paper presents a novel sparse matrix approach to efficiently perform padded convolution operations, providing theoretical insights and practical implementation for improved computational performance on CPU and GPU.

Contribution

It introduces a sparse matrix representation for convolutions with stride and padding, including an explicit formula for non-zero multiplications, and demonstrates its effectiveness in practice.

Findings

The method reduces computation by exploiting sparsity in convolutions.

Theoretical analysis provides explicit non-zero multiplication counts.

Implementation shows performance gains on CPU and GPU architectures.

Abstract

We introduce an algorithm for efficiently representing convolution with zero-padding and stride as a sparse transformation matrix, applied to a vectorized input through sparse matrix-vector multiplication (SpMV). We provide a theoretical contribution with an explicit expression for the number of non-zero multiplications in convolutions with stride and padding, offering insight into the potential for leveraging sparsity in convolution operations. A proof-of-concept implementation is presented in Python, demonstrating the performance of our method on both CPU and GPU architectures. This work contributes to the broader exploration of sparse matrix techniques in convolutional algorithms, with a particular focus on leveraging matrix multiplications for parallelization. Our findings lay the groundwork for future advancements in exploiting sparsity to improve the efficiency of convolution operations in fields such as machine learning and signal processing.