CHAOS: A Parallelization Scheme for Training Convolutional Neural Networks on Intel Xeon Phi

TL;DR

This paper introduces CHAOS, a parallelization scheme for training CNNs efficiently on Intel Xeon Phi, achieving significant speedups through thread and vector parallelism with implicit synchronization.

Contribution

CHAOS is a novel parallel training scheme supporting arbitrary synchronization order and non-instant weight updates tailored for Intel Xeon Phi systems.

Findings

Up to 103x speedup on Xeon Phi with multiple threads

14x faster than sequential Xeon E5 execution

58x faster than sequential Core i5 execution

Abstract

Deep learning is an important component of big-data analytic tools and intelligent applications, such as, self-driving cars, computer vision, speech recognition, or precision medicine. However, the training process is computationally intensive, and often requires a large amount of time if performed sequentially. Modern parallel computing systems provide the capability to reduce the required training time of deep neural networks. In this paper, we present our parallelization scheme for training convolutional neural networks (CNN) named Controlled Hogwild with Arbitrary Order of Synchronization (CHAOS). Major features of CHAOS include the support for thread and vector parallelism, non-instant updates of weight parameters during back-propagation without a significant delay, and implicit synchronization in arbitrary order. CHAOS is tailored for parallel computing systems that are accelerated with the Intel Xeon Phi. We evaluate our parallelization approach empirically using measurement techniques and performance modeling for various numbers of threads and CNN architectures. Experimental results for the MNIST dataset of handwritten digits using the total number of threads on the Xeon Phi show speedups of up to 103x compared to the execution on one thread of the Xeon Phi, 14x compared to the sequential execution on Intel Xeon E5, and 58x compared to the sequential execution on Intel Core i5.

Tables

Table 1. Table 1 : Execution times at each layer for the sequential version on the Xeon E5 using the small CNN architecture.

Layer type	Forward propagation	Back-propagation	% of total
Fully connected	40.9 s	30.9 s	1.4 %
Convolutional	3,241 s	1,438 s	93.7 %
Max pooling	188.3 s	8.2 s	3.9 %

Table 2. Table 2 : CNN architectures for experimental evaluation of CHAOS.

	Layer Type	Maps	Map Size	Neurons	Kernel Size	Weights
Large	Input	-	29x29	841	-	-
	Convolutional	20	26x26	13520	4x4	340
	Max-pooling	20	26x26	13520	1x1	-
	Convolutional	60	22x22	29040	5x5	30060
	Max-pooling	60	11x11	7260	2x2	-
	Convolutional	100	6x6	3600	6x6	216100
	Max-pooling	100	2x2	900	3x3	-
	Fully connected	-	150	150	-	135150
	Output	-	10	10	-	1510
Medium	Input	-	29x29	841	-	-
	Convolutional	20	26x26	13520	4x4	340
	Max-pooling	20	13x13	3380	2x2	-
	Convolutional	40	9x9	3240	5x5	20040
	Max-pooling	40	3x3	360	3x3	-
	Fully connected	-	150	150	-	54150
	Output	-	10	10	-	1510
Small	Input	-	29x29	841	-	-
	Convolutional	5	26x26	3380	4x4	85
	Max-pooling	5	13x13	845	2x2	-
	Convolutional	10	9x9	810	5x5	1260
	Max-pooling	10	3x3	90	3x3	-
	Fully connected	-	50	50	-	4550
	Output	-	10	10	-	510

Full text

1 Introduction

Traditionally engineers developed applications by specifying computer instructions that determined the application behavior. Nowadays engineers focus on developing and implementing sophisticated deep learning models that can learn to solve complex problems. Moreover, deep learning algorithms lecun2015nature can learn from their own experience rather than that of the engineer.

Many private and public organizations are collecting huge amounts of data that may contain useful information from which valuable knowledge may be derived. With the pervasiveness of the Internet of Things the amount of available data is getting much larger Hsu2014. Deep learning is a useful tool for analyzing and learning from massive amounts of data (also known as Big Data) that may be unlabeled and unstructured strawn2016; sharma2014; najafabadi2015. Deep learning algorithms can be found in many modern applications siri; taigman2014deepface; hadsell2009learning; wu2014deep; szegedy2015going; fox2014; jiang2016; zhang2012, such as, voice recognition, face recognition, autonomous cars, classification of liver diseases and breast cancer, computer vision, or social media.

A Convolutional Neural Network (CNN) is a variant of a Deep Neural Network (DNN) 110_deeplearning_lenet. Inspired by the visual cortex of animals, CNNs are applied to state-of-the-art applications, including computer vision and speech recognition deng2014deep. However, supervised training of CNNs is computationally demanding and time consuming, and in many cases, several weeks are required to complete a training session. Often applications are tested with different parameters, and each test requires a full session of training.

Multi-core processors Williams:2009 and in particular many-core benkner11 processing architectures, such as the NVIDIA Graphical Processing Unit (GPU) nvidia_gpu or the Intel Xeon Phi chrysos2012intel co-processor, provide processing capabilities that may be used to significantly speed-up the training of CNNs. While existing research cirecsan2011high; vrtanoski2012pattern; szegedy2015going; yadan2013multi; sainath2015deep has addressed extensively the training of CNNs using GPUs, so far not much attention is given to the Intel Xeon Phi co-processor. Beside the performance capabilities, the Xeon Phi deserves our attention because of programmability pllana09 and portability KesslerDTNRDBTP12.

In this paper, we present our parallelization scheme for training convolutional neural networks, named Controlled Hogwild with Arbitrary Order of Synchronization (CHAOS). CHAOS is tailored for the Intel Xeon Phi co-processor and exploits both the thread- and SIMD-level parallelism. The thread-level parallelism is used to distribute the work across the available threads, whereas SIMD parallelism is used to compute the partial derivatives and weight gradients in convolutional layer. Empirical evaluation of CHAOS is performed on an Intel Xeon Phi 7120 co-processor. For experimentation, we use various number of threads, different CNNs architectures, and the MNIST dataset of handwritten digits lecun2010mnist. Experimental evaluation results show that using the total number of available threads on the Intel Xeon Phi we can achieve speedups of up to $103\times$ compared to the execution on one thread of the Xeon Phi, $14\times$ compared to the sequential execution on Intel Xeon E5, and $58\times$ compared to the sequential execution on Intel Core i5. The error rates of the parallel execution are comparable to the sequential one. Furthermore, we use performance prediction to study the performance behavior of our parallel solution for training CNNs for numbers of cores that go beyond the generation of the Intel Xeon Phi that was used in this paper. The main contributions of this paper include:

• •

design and implementation of CHAOS parallelization scheme for training CNNs on the Intel Xeon Phi,

• •

performance modeling of our parallel solution for training CNNs on the Intel Xeon Phi,

• •

measurement-based empirical evaluation of CHAOS parallelization scheme,

• •

model-based performance evaluation for future architectures of the Intel Xeon Phi.

The rest of the paper is organized as follows. We discuss the related work in Section 2. Section 3 provides background information on CNNs and the Intel Xeon Phi many-core architecture. Section 4 discusses the design and implementation aspects of our parallelization scheme. The experimental evaluation of our approach is presented in Section 5. We summarize the paper in Section LABEL:sec:conclusion.

2 Related Work

In comparison to related work that target GPUs, the work related to machine learning for Intel Xeon Phi is sparse. In this section, we describe machine learning approaches that target the Intel Xeon Phi co-processor, and thereafter we discuss CNN solutions for GPUs and contrast them to our CHAOS implementation.

2.1 Machine Learning targeting Intel Xeon Phi

In this section, we discuss existing work for Support Vector Machines (SVMs), Restricted Boltzmann Machines (RBMs), sparse auto encoders and the Brain-State-in-a-Box (BSB) model.

You et al. you2014mic present a library for parallel Support Vector Machines, MIC-SVM, which facilitates the use of SVMs on many- and multi-core architectures including Intel Xeon Phi. Experiments performed on several known datasets showed up to 84x speed up on the Intel Xeon Phi compared to the sequential execution of LIBSVM chang2011libsvm. In comparison to their work, we target deep learning.

Jin et al. JinWGYH14 perform the training of sparse auto encoders and restricted Boltzmann machines on the Intel Xeon Phi 5110p. The authors reported a speed up factor of $7-10\times$ times compared to the Xeon E5620 CPU and more than $300\times$ times compared to the un-optimized version executed on one thread on the co-processor. Their work targets unsupervised deep learning of restricted Boltzmann machines and sparse auto encoders, whereas we target supervised deep learning of CNNs.

The performance gain on Intel Xeon Phi 7110p for a model called Brain-State-in-a-Box (BSB) used for text recognition is studied by Ahmed et al. in ahmed2014accelerating. The authors report about two-fold speedup for the co-processor compared to a CPU with 16 cores when parallelizing the algorithm. While both approaches target Intel Xeon Phi, our work addresses training of CNNs on the MNIST dataset.

2.2 Related Work Targeting CNNs

In this section, we will discuss CNNs solutions for GPUs in the context of computer vision (image classification). Work related to MNIST lecun2010mnist dataset is of most interest, also NORB lecun2004learning and CIFAR 10 krizhevsky2009learning is considered. Additionally, work done in speech recognition and document processing is briefly addressed. We conclude this section by contrasting the presented related work with our CHAOS parallelization scheme.

Work presented by Cireşan et al. cirecsan2011high target a CNN implementation raising the bars for the CIFAR10 (19.51% error rate), NORB (2.53% error rate) and MNIST (0.35% error rate) datasets. The training was performed on GPUs (Nvidia GTX 480 and GTX 580) where the authors managed to decrease the training time severely - up to $60\times$ compared to sequential execution on a CPU - and decrease the error rates to an, at the time, state-of-the-art accuracy level.

Later, Cireşan et al. cirecsan2012multi presented their multi-column deep neural network for classification of traffic sings. The results show that the model performed almost human-like (humans'error rate about 0.20%) on the MNIST dataset, achieving a best error rate of 0.23%. The authors trained the network on a GPU.

Vrtanoski et al. vrtanoski2012pattern use OpenCL for parallelization of the back-propagation algorithm for pattern recognition. They showed a significant cost reduction, a maximum speedup of $25.8\times$ was achieved on an ATI 5870 GPU compared to a Xeon W3530 CPU when training the model on the MNIST dataset.

The ImageNet challenge aims to evaluate algorithms for large-scale object detection and image classification based on the ImageNet dataset. Krizhevsky et al. krizhevsky2012imagenet joined the challenge and reduced the error rate of the test set to 15.3% from the second best 26.2% using a CNN with 5 convolutional layers. For the experiments, two GPUs (Nvidia GTX 580) were used only communicating in certain layers. The training lasted for 5 to 6 days.

In a later challenge, ILSVRC 2014, a team from Google entered the competition with GoogleNet, a 22-layer deep CNN and won the classification challenge with a 6.67% error rate. The training was carried out on CPUs. The authors state that the network could be trained on GPUs within a week, illuminating the limited amount of memory to be one of the major concerns szegedy2015going.

Yadan et al. yadan2013multi used multiple GPUs to train CNNs on the ImageNet dataset using both data- and model-parallelism, i.e. either the input space is divided into mini-batches where each GPU train its own batch (data parallelism) or the GPUs train one sample together (model parallelism). There is no direct comparison with the training time on CPU, however, using 4 GPUs (Nvidia Titan) and model- and data-parallelism, the network was trained for 4.8 days.

Song et al. song2014deep constructed a CNN to recognize face expressions and developed a smart-phone app in which the user can capture a picture and send it to a server hosting the network. The network, predicts a face expression and sends the result back to the user. With the help of GPUs (Nvidia Titan), the network was trained in a couple of hours on the ImageNet dataset.

Scherer et al. scherer2010accelerating accelerated the large-scale neural networks with parallel GPUs. Experiments with the NORB dataset on an Nvidia GTX 285 GPU showed a maximal speedup of $115\times$ compared to a CPU implementation (Core i7 940). After training the network for 360 epochs, an error rate of 8.6% was achieved.

Cireşan et al. cirecsan2011committee combined multiple CNNs to classify German traffic signs and achieved a 99.15% recognition rate (0.85 % error rate). The training was performed using an Intel Core i7 and 4 GPUs (2 x GTX 480 and 2 x GTX 580).

More recently Abadi et al. tensorflow2015whitepaper presented TensorFlow, a system for expressing and executing machine learning algorithms including training deep neural network models.

Researchers have also found CNNs successful for speech tasks. Large vocabulary continuous speech recognition deals with translation of continuous speech for languages with large vocabularies. Sainath et al. sainath2015deep investigated the advantages of CNNs performing speech recognition tasks and compared the results with previous DNN approaches. Results indicated on a 12-14% relative improvement of word error rates compared to a DNN trained on GPUs.

Chellapilla et al. chellapilla2006high investigated GPUs (Nvidia Geforce 7800 Ultra) for document processing on the MNIST dataset and achieved a $4.11\times$ speed up compared to the sequential execution a Intel Pentium 4 CPU running at 2.5 GHz clock frequency.

In contrast to CHAOS, these studies target training of CNNs using GPUs, whereas our approach addresses training of CNNs on the MNIST dataset using the Intel Xeon Phi co-processor. While there are several review papers (such as, bahrampour2015comparative; shi2016benchmarking; tabik2017snapshot) and on-line articles (such as, murphy2016review) that compare existing frameworks for parallelization of training CNN architectures, we focus on detailed analysis of our proposed parallelization approach using measurement techniques and performance modeling. We compare the performance improvement achieved with CHAOS parallelization scheme to the sequential version executed on Intel Xeon Phi, Intel Xeon E5 and Intel Core i5 processor.

3 Background

In this section, we first provide some background information related to the neural networks focusing on convolutional neural networks, and thereafter we provide some information about the architecture of the Intel Xeon Phi.

3.1 Neural Networks

A Convolutional Neural Network is a variant of a Deep Neural Network, which introduces two additional layer types: convolutional layers and pooling layers. The mammal visual processing system is hierarchical (deep) in nature. Higher level features are abstractions of lower level ones. E.g. to understand speech, waveforms are translated through several layers until reaching a linguistic level. A similar analogy can be drawn for images, where edges and corners are lower level abstractions translated into more spatial patterns on higher levels. Moreover, it is also known that the animal cortex consists of both simple and complex cells firing on certain visual inputs in their receptive fields. Simple cells detect edge-like patterns whereas complex cells are locally invariant, spanning larger receptive fields. These are the very fundamental properties of the animal brain inspiring DNNs and CNNs.

In this section, we first describe the DNNs and the Forward- and Back-propagation, thereafter we introduce the CNNs.

3.1.1 Deep Neural Networks

The architecture of a DNN consists of multiple layers of neurons. Neurons are connected to each other through edges (weights). The network can simply be thought of as a weighted graph; a directed acyclic graph represents a feed-forward network. The depth and breadth of the network differs as may the layer types. Regardless of the depth, a network has at least one input and one output layer. A neuron has a set of incoming weights, which have corresponding outgoing edges attached to neurons in the previous layer. Also, a bias term is used at each layer as an intercept term. The goal of the learning process is to adjust the network weights and find a global minimum by reducing the overall error, i.e. the deviation between the predicted and the desired outcome of all the samples. The resulting weight parameters can thereafter be used to make predictions of unseen inputs ng2011ufldl.

3.1.2 Forward Propagation

DNNs can make predictions by forward propagating an input through the network. Forward propagation proceeds by performing calculations at each layer until reaching the output layer, which contains a vector representing the prediction. For example, in image classification problems, the output layer contains the prediction score that indicates the likelihood that an image belongs to a category agibansky; ng2011ufldl.

The forward propagation starts from a given input layer, then at each layer the activation for a neuron is activated using the equation $y^{l}_{i}=\sigma(x^{l}_{i})+I^{l}_{i}$ where $y^{l}_{i}$ is the output value of neuron $i$ at layer $l$, $x^{l}_{i}$ is the input value of the same neuron, and $\sigma$ (sigmoid) is the activation function. $I^{l}_{i}$ is used for the input layer when there is no previous layer. The goal of the activation function is to return a normalized value (sigmoid return [0,1] and tanh is used in cases where the desired return values are [-1,1]). The input $x^{l}_{i}$ can be calculated as $x^{l}_{i}=\sum_{j}(w^{l}_{ji}y^{l-1}_{j})$ where $w^{l}_{ji}$ denotes the weight between neuron $i$ in the current layer $l$, and $j$ in the previous layer, and $y^{l-1}_{j}$ the output of the $j$th neuron at the previous layer. This process is repeated until reaching the output layer. At the output layer, it is common to apply a soft max function, or similar, to squash the output vector and hence derive the prediction.

3.1.3 Back-Propagation

Back-propagation is the process of propagating errors, i.e. the loss calculated as the deviation between the predicted and the desired output, backward in the network, by adjusting the weights at each layer. The error and partial derivatives $\delta^{l}_{i}$ are calculated at the output layer based on the predicted values from forward propagation and the labeled value (the correct value). At each layer, the relative error of each neuron is calculated and the weight parameters are updated based on how much the neuron participated in the faulty prediction. The equation: $\dfrac{\delta E}{\delta y^{l}_{i}}=\sum{w^{l}_{ij}\dfrac{\delta E}{\delta x^{l+1}_{j}}}$ denotes that the partial derivative of neuron $i$ at the current layer $l$ is the sum of the derivatives of connected neurons at the next layer multiplied with the weights, assuming $w^{l}$ denotes the weights between the maps. Additionally, a decay is commonly used to control the impact of the updates, which is omitted in the above calculations. More concretely, the algorithm can be thought of as updating the layer's weights based on ”how much it was responsible for the errors in the output” agibansky; ng2011ufldl.

3.1.4 Convolutional Neural Networks

A Convolutional Neural Network is a multi-layer model constructed to learn various levels of representations where higher level representations are described based on the lower level ones schmidhuber2015deep. It is a variant of deep neural network that introduces two new layer types: convolutional and pooling layers.

The convolutional layer consists of several feature maps where neurons in each map connect to a grid of neurons in maps in the previous layer through overlapping kernels. The kernels are tiled to cover the whole input space. The approach is inspired by the receptive fields of the mammal visual cortex. All neurons of a map extract the same features from a map in the previous layer as they share the same set of weights.

Pooling layers intervene convolutional layers and have shown to lead to faster convergence. Each neuron in a pooling layer outputs the (maximum/average) value of a partition of neurons in the previous layer, and hence only activates if the underlying grid contains the sought feature. Besides from lowering the computational load, it also enables position invariance and down samples the input by a factor relative to the kernel size lecun1998gradient.

Figure 1 shows LeNet-5 that is an example of a Convolutional Neural Network. Each layer of convolution and pooling (that is a specific method of sub-sampling used in LeNet) comprise several feature maps. Neurons in the feature map cover different sub-fields of the neurons from the previous layer. All neurons in a map share the same weight parameters, therefore they extract the same features from different parts of the input from the previous layers.

CNNs are commonly constructed similarly to the LeNet-5, beginning with an input layer, followed by several convolutional/pooling combinations, ending with a fully connected layer and an output layer lecun1998gradient. Recent networks are much deeper and/or wider, for instance, the GoogleNet szegedy2015going consists of 22 layers.

Various implementations target the Convolutional Neural Networks, such as: EbLearn at New York University and Caffe at Berkeley. As a basis for our work we selected a project developed by Cireşan dciresan. This implementation targets the MNIST dataset of handwritten digits, and has the possibility to dynamically configure the definition of layers, the activation function and the connection types using a configuration file.

3.2 Parallel Systems accelerated with Intel®Xeon Phi™

Figure 2 depicts an overview of the Intel Xeon Phi (codenamed Knights Corner) architecture. It is a many-core shared-memory co-processor, which runs a lightweight Linux operating system that offers the possibility to communicate with it over ssh. The Xeon Phi offers two programming models:

offload - parts of the applications running on the host are offloaded to the co-processor 2. 2.

native - the code is compiled specifically for running natively on the co-processor. The code and all the required libraries should be transferred on the device. In this paper, we focus on the native mode.

The Intel Xeon Phi (type 7120P used in this paper) comprises 61 x86 cores, each core runs at 1.2 GHz base frequency, and up to 1.3GHz on max turbo frequency chrysos2012intel. Each core can switch between four hardware threads in a round-robin manner, which amounts to a total of 244 threads per co-processor. Theoretically, the co-processor can deliver up to one teraFLOP/s of double precision performance, or two teraFLOP/s of single precision performance. Each core has its own L1 (32KB) and L2 (512KB) cache. The L2 cache is kept fully coherent by a global distributed tag-directory (TD). The cores are connected through a bidirectional ring bus interconnect, which forms a unified shared L2 cache of 30.5MB. In addition to the cores, there are 16 memory channels that in theory offer a maximum memory bandwidth of 352GB/s. The GDDR memory controllers provide direct interface to the GDDR5 memory, and the PCIe Client Logic provides direct interface to the PCIe bus.

Efficient usage of the available vector processing units of the Intel Xeon Phi is essential to fully utilize the performance of the co-processor TianSPGKMCP13. Through the 512-bit wide SIMD registers it can perform 16 (16 wide $\times$ 32 bit) single-precision or 8 (8 wide $\times$ 64 bit) double-precision operations per cycle.

The performance capabilities of the Intel Xeon Phi are discussed and investigated empirically by different researches within several domain applications dokulil13; lu2013optimizing; DNAxphi; teodoro2013comparative; leung2013investigating; CPE4037.

4 Our Parallelization Scheme for Training Convolutional Neural Networks on Intel Xeon Phi

The parallelism can be either divided data-wise, i.e. threads process several inputs concurrently, or model-wise, i.e. several threads share the computational burden of one input. Whether one approach can be advantageous over the other mainly depends on the synchronization overhead of the weight vectors and how well it scales with the number of processing units.

In this section, we first discuss the design aspects of our parallelization scheme for training convolutional neural networks. Thereafter, we discuss the implementation aspects that allow full utilization of the Intel Xeon Phi co-processor.

4.1 Design Aspects

On-line stochastic gradient descent has the advantage of instant updates of weights for each sample. However, the sequential nature of the algorithm yields impediments as the number of multi- and many-core platforms are emerging. We consider different existing parallelization strategies for stochastic gradient descent:

**Strategy A: Hybrid - ** uses both data- and model parallelism, such that data parallelism is applied in convolutional layers, and the model parallelism is applied in fully connected layers krizhevsky2014one.

**Strategy B: Averaged Stochastic Gradient - ** divides the input into batches and feeds each batch to a node. This strategy proceeds as follows: (1) Initialize the weights of the learner by randomization; (2) Split the training data into n equal chunks and send them to the learners; (3) each learner process the data and calculates the weight gradients for its batch; (4) send the calculated gradients back to the master; (5) the master computes and updates the new weights; and (6) the master sends the new weights to the nodes and a new iteration begins de2012parallelization. The convergence speed is slightly worse than for the sequential approach, however the training time is heavily reduced.

**Strategy C: Delayed Stochastic Gradient - ** suggests updating the weight parameters in a round-robin fashion by the workers. One solution is splitting the samples by the number of threads, and let each thread work on its own distinct chunk of samples, only sharing a common weight vector. Threads are only allowed to update the weight vector in a round-robin fashion, and hence each update will be delayed ZinkevichSL09.

**Strategy D: HogWild! - ** is a stochastic gradient descent without locks. The approach is applicable for sparse optimization problems (threads/core updates do not conflict much) recht2011hogwild.

In this paper, we introduce Controlled Hogwild with Arbitrary Order of Synchronization (CHAOS), a parallelization scheme that can exploit both thread- and SIMD-level parallelism available on Intel Xeon Phi. CHAOS is a data-parallel controlled version of HogWild! with delayed updates, which combines parts of strategies A-D. The key aspects of CHAOS are:

• •

Thread parallelism - The overview of our parallelization scheme is depicted in Figure 3. Initially for as many threads as there are available network instances are created, which share weight parameters, whereas to support concurrent processing of images some variables are private to each thread. After the initialization of CNNs and images is done, the process of training starts. The major steps of an epoch include: Training, Validation and Testing. The first step, Training, proceeds with each worker picking an image, forward propagates it through the network, calculates the error, and back-propagates the partial derivatives, adjusting the weight parameters. Since each worker picks a new image from the set, other workers do not have to wait for significantly slow workers. After Training, each worker participates in Validation and Testing evaluating the prediction accuracy of the network by predicting images in the validation and test set accordingly. Adoption of data parallelism was inspired by Krizhevsky krizhevsky2014one, promoting data parallelism for convolutional layers as they are computationally intensive.

• •

Controlled HogWild - during the back-propagation the shared weights are updated after each layer's computations (a technique inspired by ZinkevichSL09), whereas the local weight parameters are updated instantly (a technique inspired by recht2011hogwild), which means that the gradients are calculated locally first then shared with other workers. However, the update to the global gradients can be performed at any time, which means that there is no need to wait for other workers to finish their updates. This technique, which we refer to as non-instant updates of weight parameters without significant delay, allows us to avoid unnecessary cache line invalidation and memory writes.

• •

Arbitrary Order of Synchronization - There is no need for explicit synchronization, because all workers share weight parameter. However, an implicit synchronization is performed in an arbitrary order because writes are controlled by a first-come-first schedule and reads are performed on demand.

The main goal of CHAOS is to minimize the time spent in the convolutional layers, which can be done through data parallelism, adapting the knowledge presented in strategy A. In strategy B, the synchronization is performed because of averaging worker's gradient calculations. Since work is distributed, computations are performed on stale parameters. The strategy can be applied in distributed and non-distributed settings. The division of work over several distributed workers was adapted in CHAOS. In strategy C, the updates are postponed using a round-robin-fashion where each thread gets to update when it is its turn. The difference compared to strategy B is that instances train on the same set of weights and no averaging is performed. The advantage is that all instances train on the same weights. The disadvantage of this approach is the delayed updates of the weight parameters as they are performed on stale data. Training on shared weights and delaying the updates are adopted in CHAOS. Strategy D presents a lock-free approach of updating the weight parameters, updates are performed instantly without any locks. Our updates are not instant, however, after computing the gradients there is nothing prohibiting a worker contributing to the shared weights, the notion of instant inspired CHAOS.

4.2 Implementation Aspects

The main goal is to utilize the many cores of the Intel Xeon Phi co-processor efficiently to lower the training time (execution time) of the selected CNN algorithm, at the same time maintaining low deviation in error rates, especially on the test set. Moreover, the quality of the implementation is verified using errors and error rates on the validation and test set.

In the sequential version, only minor modifications of the original version were performed. Mainly, we added a Reporter class to serialize execution results. The instrumentation should not add any time penalties in practice. However, if these penalties occur in the sequential version they are likely to imply corresponding penalties in the parallel version, therefore it should not impact the results.

The main goal of the parallel version is to lower the execution time of the sequential implementation and to scale well with the number of processing units on the co-processor. To facilitate this, it is essential to fully consider the characteristics of the underlying hardware. From results derived in the sequential execution we found the hotspots of the application to be predominantly the convolutional layers. The time spent in both forward- and back-propagation is about 94% of the total time of all layers (up to 99% for the larger network), which is depicted in the Table 1.

In our proposed strategy, a set of $N$ network instances are created and assigned to $T$ threads. We assume $T==N$, i.e. one thread per network instance. $T$ threads are spawned, each responsible for its own instance.

The overview of the algorithm is shown in Fig. 3. In Fig. 4 the training, testing and back-propagation phase are shown in details. Training (see Fig. 4(a)) picks an image, forward propagates it, determines the loss and back-propagates the partial derivatives (deltas) in the network - this process is done simultaneously by all workers, each worker processing one image. Each worker participating in testing (see Fig. 4(b)), picks an image, forward propagates it and then collects errors and error rates. The results are cumulated for all threads. Perhaps the most interesting part is the back-propagation (see Fig. 4(c)). The shared weights are used when propagating the deltas, however, before updating the weight gradients, the pointers are set to the local weights. Thereafter the algorithm proceeds by updating the local weights first. When a worker has contributions to the global weights it can update in a controlled manner, avoiding data races. Updates immediately affect other workers in their training process. Hence the update is delayed slightly, to decrease the invalidation of cache lines, yet almost instant and workers do not have to wait for a longer period before contributing with their knowledge.

To see why delays are important, consider the following scenario: If training several network instances concurrently, they share the same weight vectors, other variables are thread private. The major consideration lies in the weight updates. Let $W^{j}_{l}$ be the $j$-th weight on the $l$-th layer. In accordance with the current implementation, a weight is updated several times since neurons in a map (on the same layer) share the same weights, and the kernel is shifted over the neurons. Further assume that several threads work on the same weight $W^{j}_{l}$ at some point in time. Even if other threads only read the weights, their local data, as saved in the Level 2 cache, will be invalidated and a re-fetch is required to assert their integrity. This happens because cache lines are shared between cores. The approach of slightly delaying the updates and forcing one thread to update in atomicity leads to fewer invalidations. Still a major disadvantages is that the shared weights does not infer any data locality (data cannot retain completely in Level 2 cache for a longer period).

To further decrease the time spent in convolutional layers, loops were vectorized to facilitate the vector processing unit of the co-processor. Data was allocated using $\_mm\_malloc()$ with $64$ byte alignment increasing the accuracy of memory requests. The vectorization was achieved by adding $\#pragma~{}omp~{}simd$ instructions and explicitly informing the compiler of the memory alignment using $\_\_assume\_aligned()$. Some unnecessary overhead is added through the lack of data alignment of the deltas and weights. The computations of partial derivatives and weight gradients in the convolutional layers are performed in a SIMD way, which allows efficient utiliziation of the 512 bit wide vector processing units of the Intel Xeon Phi. An extract from the vectorization report (see Listing 1), for the updates of partial derivatives in the convolutional layer shows an estimated potential speed up of $3.98\times$ compared to the scalar loop.

Further algorithmic optimizations were performed. For example: (1) The images are loaded into a pre-allocated memory instead of allocating new memory when requesting an image; (2) Hardware pre-fetching was applied to mitigate the shortcomings of the in-order-execution scheme. Pre-fetching loads data to L2 cache to make it available for future computations; (3) Letting workers pick images instead of assigning images to workers, allow for a smaller overhead at the end of a work-sharing construct; (4) The number of locks are minimized as far as possible; (5) We made most of the variables thread private to achieve data locality.

The training phase was distributed through thread parallelism, dividing the input space over available workers. CHAOS uses the vector processing units to improve performance and tries to retain local variables in local cache as far as possible. The delayed updates decrease the invalidation of cache lines. Since weight parameters are shared among threads, there is a possibility that data can be fetched from another core's cache instead of main memory, reducing the wait times. Also, the memory was aligned to 64 bytes and unnecessary system calls were removed from the parallel work.

5 Evaluation

In this section, we first describe the experimentation environment used for evaluation of our CHAOS parallelization scheme. Thereafter, we describe the development of a performance model for CHAOS. Finally we discuss the obtained results with respect to scalability, speedup, and prediction accuracy.

5.1 Experimental Setup

In this study, OpenMP was selected to facilitate the utilization of thread- and SIMD-parallelism available in the Intel Xeon Phi co-processor. C++ programming language is used for algorithm implementation. The Intel Compiler 15.0.0 was used for native compilation of the application for the co-processor, whereas the $O3$ level was used for optimization.

System Configuration - To evaluate our approach we use an Intel Xeon Phi accelerator that comprises 61 cores that run at 1.2 GHz. For evaluation 1, 15, 30, 60, 120, 180, 240, and 244 threads of the co-processor were used. Each thread was responsible for one network instance. For comparison, we use two general purpose CPUs, including the Intel Xeon E5-s695v2 that runs at 2.4 GHz clock frequency, and the Intel Core i5 661 that runs at 3.33GHz clock frequency.

Data Set - To evaluate our approach, the MNIST lecun2010mnist dataset of handwritten digits is used. In total the MNIST dataset comprises 70000 images, 60000 of which are used for training/validation and the rest for testing.

CNN Architectures - Three different CNN architectures were used for evaluation, small, medium and large. The small and medium architecture were trained for 70 epochs, and the large one for 15 epochs, using a starting decay (eta) of 0.001 and factor of 0.9. The small and medium network consist of seven layers in total (one input layer, two convolutional layers, two max-poling layers, one fully connected layer and the output layer). The difference between these two networks is in the number of feature maps per layer and the number of neurons per map. For example, the first convolutional layer of the small network has five feature maps and 3380 neurons, whereas the first convolutional layer of the medium network has 20 feature maps and 13520 neurons. The large network differs from the small and the medium network in the number of layers as well. In total, there are nine layers, one input layer, three convolutional layers, three max-pooling layers, one fully connected layer and the output layer. Detailed information (including the number and the size of feature maps, neurons, the size of the kernels and the weights) about the considered architectures is listed in Table 2.

To address the variability in performance measurements we have repeated the execution of each parallel configuration for three times.

5.2 Performance Model

A performance model perfmod enables us to reason about the behavior of an implementation in future execution contexts. Our performance model for CHAOS implementation can predict the performance for numbers of threads that go beyond the number of hardware threads supported in the Intel Xeon Phi model that we used for evaluation. Additionally, it can predict the performance of different CNN architectures with various number of images and epochs.

The goal is to construct a parametrized model with the following parameters $ep,i,it$ and $p$, where $ep$ stands for the number of epochs, $i$ indicates the number of images in the training/validation set, $it$ stands for the number of images in the test set, and $p$ is the number of processing units. Table LABEL:tab:perf-model-parameters lists the full set of variables used in our performance model, some of which are hardware dependent and some others are independent of the underlying hardware. Each variable is either measured, calculated, constant, or parameter in the model. Listing LABEL:listing:performance-prediction shows the formula used for our performance prediction model.