dMath: Distributed Linear Algebra for DL

TL;DR

dMath is a distributed linear algebra library optimized for deep learning that offers scalable performance, easy-to-use primitives, and efficient memory management for building large-scale neural network applications.

Contribution

The paper introduces dMath, a novel library providing scalable distributed primitives and algorithms tailored for deep learning, with advanced memory management techniques.

Findings

Achieves leading scaling performance in DL workloads.

Supports a variety of domain-specific algorithms.

Enables rapid development of scalable DNN applications.

Abstract

The paper presents a parallel math library, dMath, that demonstrates leading scaling when using intranode, internode, and hybrid-parallelism for deep learning (DL). dMath provides easy-to-use distributed primitives and a variety of domain-specific algorithms including matrix multiplication, convolutions, and others allowing for rapid development of scalable applications like deep neural networks (DNNs). Persistent data stored in GPU memory and advanced memory management techniques avoid costly transfers between host and device. dMath delivers performance, portability, and productivity to its specific domain of support.