Towards a Scalable and Distributed Infrastructure for Deep Learning Applications

TL;DR

This paper introduces Phylanx, a scalable distributed deep learning infrastructure that translates Python code into efficient multi-node execution using fine-grained parallelism and task-based runtime systems.

Contribution

It presents a novel framework that enhances distributed deep learning by enabling fine-grained inter-node communication and efficient execution of Python code.

Findings

Phylanx improves scalability for deep learning workloads.

It enables efficient fine-grained communication across nodes.

The framework leverages C++ parallelism and concurrency libraries.

Abstract

Although recent scaling up approaches to training deep neural networks have proven to be effective, the computational intensity of large and complex models, as well as the availability of large-scale datasets, require deep learning frameworks to utilize scaling out techniques. Parallelization approaches and distribution requirements are not considered in the preliminary designs of most available distributed deep learning frameworks, and most of them still are not able to perform effective and efficient fine-grained inter-node communication. We present Phylanx that has the potential to alleviate these shortcomings. Phylanx offers a productivity-oriented frontend where user Python code is translated to a futurized execution tree that can be executed efficiently on multiple nodes using the C++ standard library for parallelism and concurrency (HPX), leveraging fine-grained threading and an active messaging task-based runtime system.