BigDL: A Distributed Deep Learning Framework for Big Data

TL;DR

BigDL is a distributed deep learning framework integrated with Apache Spark, enabling scalable, end-to-end deep learning applications directly on big data platforms for industry use.

Contribution

It introduces a novel distributed training approach on Spark's functional compute model, facilitating seamless integration of deep learning with big data processing.

Findings

Used successfully in industry for production deep learning applications.

Enables direct processing of production data within Spark clusters.

Supports end-to-end data analysis and deep learning pipelines.

Abstract

This paper presents BigDL (a distributed deep learning framework for Apache Spark), which has been used by a variety of users in the industry for building deep learning applications on production big data platforms. It allows deep learning applications to run on the Apache Hadoop/Spark cluster so as to directly process the production data, and as a part of the end-to-end data analysis pipeline for deployment and management. Unlike existing deep learning frameworks, BigDL implements distributed, data parallel training directly on top of the functional compute model (with copy-on-write and coarse-grained operations) of Spark. We also share real-world experience and "war stories" of users that have adopted BigDL to address their challenges(i.e., how to easily build end-to-end data analysis and deep learning pipelines for their production data).