A Highly Parallel FPGA Implementation of Sparse Neural Network Training

TL;DR

This paper presents a reconfigurable FPGA architecture for sparse neural network training and inference, enabling efficient on-chip processing with reduced complexity and greater hyperparameter exploration.

Contribution

It introduces a highly parallel, reconfigurable FPGA design for sparse neural networks that supports on-chip training and inference with structured sparsity.

Findings

Achieved efficient FPGA implementation of sparse neural networks

Demonstrated reconfigurability to balance resource use and training speed

Enabled extensive hyperparameter exploration on-chip

Abstract

We demonstrate an FPGA implementation of a parallel and reconfigurable architecture for sparse neural networks, capable of on-chip training and inference. The network connectivity uses pre-determined, structured sparsity to significantly reduce complexity by lowering memory and computational requirements. The architecture uses a notion of edge-processing, leading to efficient pipelining and parallelization. Moreover, the device can be reconfigured to trade off resource utilization with training time to fit networks and datasets of varying sizes. The combined effects of complexity reduction and easy reconfigurability enable significantly greater exploration of network hyperparameters and structures on-chip. As proof of concept, we show implementation results on an Artix-7 FPGA.