Enabling Massive Deep Neural Networks with the GraphBLAS

TL;DR

This paper demonstrates how the GraphBLAS library enables efficient training and inference of large, sparse deep neural networks by leveraging sparse matrix operations, resulting in significant memory and performance benefits.

Contribution

It introduces a method to rewrite DNN computations using GraphBLAS, enabling high-performance sparse matrix operations for large neural networks.

Findings

Sparse GraphBLAS implementation outperforms dense BLAS as matrices become sparser.

Memory savings with sparse matrices are significant for large DNNs.

Performance improves notably for larger, sparser weight matrices.

Abstract

Deep Neural Networks (DNNs) have emerged as a core tool for machine learning. The computations performed during DNN training and inference are dominated by operations on the weight matrices describing the DNN. As DNNs incorporate more stages and more nodes per stage, these weight matrices may be required to be sparse because of memory limitations. The GraphBLAS.org math library standard was developed to provide high performance manipulation of sparse weight matrices and input/output vectors. For sufficiently sparse matrices, a sparse matrix library requires significantly less memory than the corresponding dense matrix implementation. This paper provides a brief description of the mathematics underlying the GraphBLAS. In addition, the equations of a typical DNN are rewritten in a form designed to use the GraphBLAS. An implementation of the DNN is given using a preliminary GraphBLAS C library. The performance of the GraphBLAS implementation is measured relative to a standard dense linear algebra library implementation. For various sizes of DNN weight matrices, it is shown that the GraphBLAS sparse implementation outperforms a BLAS dense implementation as the weight matrix becomes sparser.