Comparative Study of Deep Learning Software Frameworks

TL;DR

This paper compares five deep learning frameworks across extensibility, hardware utilization, and speed, providing insights into their performance on various architectures and hardware setups.

Contribution

It offers a comprehensive empirical comparison of Caffe, Neon, TensorFlow, Theano, and Torch, highlighting their strengths and weaknesses in different deep learning tasks.

Findings

Torch is best for CPU-based deep architectures.

Theano and Torch excel in GPU performance for large networks.

TensorFlow is highly flexible but less performant currently.

Abstract

Deep learning methods have resulted in significant performance improvements in several application domains and as such several software frameworks have been developed to facilitate their implementation. This paper presents a comparative study of five deep learning frameworks, namely Caffe, Neon, TensorFlow, Theano, and Torch, on three aspects: extensibility, hardware utilization, and speed. The study is performed on several types of deep learning architectures and we evaluate the performance of the above frameworks when employed on a single machine for both (multi-threaded) CPU and GPU (Nvidia Titan X) settings. The speed performance metrics used here include the gradient computation time, which is important during the training phase of deep networks, and the forward time, which is important from the deployment perspective of trained networks. For convolutional networks, we also report how each of these frameworks support various convolutional algorithms and their corresponding performance. From our experiments, we observe that Theano and Torch are the most easily extensible frameworks. We observe that Torch is best suited for any deep architecture on CPU, followed by Theano. It also achieves the best performance on the GPU for large convolutional and fully connected networks, followed closely by Neon. Theano achieves the best performance on GPU for training and deployment of LSTM networks. Caffe is the easiest for evaluating the performance of standard deep architectures. Finally, TensorFlow is a very flexible framework, similar to Theano, but its performance is currently not competitive compared to the other studied frameworks.