The Potential of the Intel Xeon Phi for Supervised Deep Learning

TL;DR

This paper explores the use of the Intel Xeon Phi coprocessor for accelerating supervised deep learning of CNNs, demonstrating significant speedups and developing a performance model to optimize training.

Contribution

It introduces the CHAOS parallelization scheme for CNN training on the Xeon Phi and provides an empirical and theoretical analysis of its performance.

Findings

103.5x speedup with 244 threads on large CNNs

Effective utilization of thread- and SIMD-parallelism

Performance model to predict and optimize training speed

Abstract

Supervised learning of Convolutional Neural Networks (CNNs), also known as supervised Deep Learning, is a computationally demanding process. To find the most suitable parameters of a network for a given application, numerous training sessions are required. Therefore, reducing the training time per session is essential to fully utilize CNNs in practice. While numerous research groups have addressed the training of CNNs using GPUs, so far not much attention has been paid to the Intel Xeon Phi coprocessor. In this paper we investigate empirically and theoretically the potential of the Intel Xeon Phi for supervised learning of CNNs. We design and implement a parallelization scheme named CHAOS that exploits both the thread- and SIMD-parallelism of the coprocessor. Our approach is evaluated on the Intel Xeon Phi 7120P using the MNIST dataset of handwritten digits for various thread counts and CNN architectures. Results show a 103.5x speed up when training our large network for 15 epochs using 244 threads, compared to one thread on the coprocessor. Moreover, we develop a performance model and use it to assess our implementation and answer what-if questions.