Nerva: a Truly Sparse Implementation of Neural Networks

TL;DR

Nerva is a new C++ library that efficiently implements sparse neural networks, reducing training time and memory usage without sacrificing accuracy, and provides a Python interface for ease of use.

Contribution

Nerva introduces a truly sparse neural network implementation using MKL, eliminating binary masks and significantly improving efficiency over existing frameworks.

Findings

Achieves 4x faster training at 99% sparsity compared to PyTorch.

Reduces memory usage while maintaining accuracy.

Supports a Python interface similar to popular frameworks.

Abstract

We introduce Nerva, a fast neural network library under development in C++. It supports sparsity by using the sparse matrix operations of Intel's Math Kernel Library (MKL), which eliminates the need for binary masks. We show that Nerva significantly decreases training time and memory usage while reaching equivalent accuracy to PyTorch. We run static sparse experiments with an MLP on CIFAR-10. On high sparsity levels like $99\%$, the runtime is reduced by a factor of $4\times$ compared to a PyTorch model using masks. Similar to other popular frameworks such as PyTorch and Keras, Nerva offers a Python interface for users to work with.