SparsePixels: Efficient Convolution for Sparse Data on FPGAs

TL;DR

SparsePixels introduces an FPGA framework for sparse CNN inference that significantly reduces latency by focusing computation on active pixels, enabling faster processing of sparse image data with minimal performance loss.

Contribution

The paper presents SparsePixels, a novel FPGA-based framework for sparse convolution that efficiently processes only active pixels, drastically reducing inference latency for sparse data.

Findings

Achieves 73x speedup over dense CNNs on FPGA for sparse data

Uses less than 1% of input pixels for computation, saving resources

Maintains high accuracy with minimal performance loss

Abstract

Inference of standard convolutional neural networks (CNNs) on FPGAs often incurs high latency and a long initiation interval due to the deep nested loops required to densely convolve every input pixel regardless of its feature value. However, input features can be spatially sparse in some image data, where semantic information may occupy only a small fraction of the pixels and most computation would be wasted on empty regions. In this work, we introduce SparsePixels, a framework that implements sparse convolution on FPGAs by selectively retaining and computing on a small subset of active pixels while ignoring the rest. We show that, for identifying neutrino interactions in naturally sparse LArTPC images with 4k pixels, a standard CNN with a compact size of 4k parameters incurs an inference latency of 48.665 $\mu$s on an FPGA, whereas a sparse CNN of the same base architecture, computing on less than 1% of the input pixels, achieves a $\times 73$ speedup to 0.665 $\mu$s with resource utilization well within on-chip budgets, trading only a small percent-level performance loss. This work aims to benefit future algorithm development for efficient data readout in modern experiments with latency requirements of microseconds or below.