SparkNet: Training Deep Networks in Spark

TL;DR

SparkNet is a framework that enables efficient training of deep neural networks in Spark clusters by addressing communication challenges, providing scalability, and maintaining compatibility with existing models, thus significantly reducing training time.

Contribution

It introduces a Spark-based deep learning framework with a simple parallelization scheme, high scalability, and ease of deployment, compatible with Caffe models and designed for high-latency environments.

Findings

SparkNet scales well with cluster size.

High-latency communication is tolerated effectively.

Benchmarking on ImageNet shows competitive performance.

Abstract

Training deep networks is a time-consuming process, with networks for object recognition often requiring multiple days to train. For this reason, leveraging the resources of a cluster to speed up training is an important area of work. However, widely-popular batch-processing computational frameworks like MapReduce and Spark were not designed to support the asynchronous and communication-intensive workloads of existing distributed deep learning systems. We introduce SparkNet, a framework for training deep networks in Spark. Our implementation includes a convenient interface for reading data from Spark RDDs, a Scala interface to the Caffe deep learning framework, and a lightweight multi-dimensional tensor library. Using a simple parallelization scheme for stochastic gradient descent, SparkNet scales well with the cluster size and tolerates very high-latency communication. Furthermore, it is easy to deploy and use with no parameter tuning, and it is compatible with existing Caffe models. We quantify the dependence of the speedup obtained by SparkNet on the number of machines, the communication frequency, and the cluster's communication overhead, and we benchmark our system's performance on the ImageNet dataset.