Atari games and Intel processors

TL;DR

This paper explores the use of CPU-based asynchronous reinforcement learning algorithms with convolutional neural networks for Atari games, analyzing their convergence and performance using Intel's Math Kernel Library and TensorFlow.

Contribution

It demonstrates the application of asynchronous RL algorithms with CNNs on CPUs and analyzes their convergence behavior, integrating Intel's MKL and TensorFlow frameworks.

Findings

Asynchronous RL algorithms are well-suited for CPU computations.

Convolutional neural networks can effectively learn strategies in Atari games.

Analysis of asynchronous computations impacts on RL convergence.

Abstract

The asynchronous nature of the state-of-the-art reinforcement learning algorithms such as the Asynchronous Advantage Actor-Critic algorithm, makes them exceptionally suitable for CPU computations. However, given the fact that deep reinforcement learning often deals with interpreting visual information, a large part of the train and inference time is spent performing convolutions. In this work we present our results on learning strategies in Atari games using a Convolutional Neural Network, the Math Kernel Library and TensorFlow 0.11rc0 machine learning framework. We also analyze effects of asynchronous computations on the convergence of reinforcement learning algorithms.

Tables

Table 1. Table 1: Description of the hyperparameters of the algorithm.

parameter	default value	description
learning rate	0.001	step size for the optimization algorithm
batch size	128	number of training examples in a training batch
frame history	4	the number of consecutive frames to take into consideration while evaluating the current state of the game
local time max	5	number of consecutive data points to memorize before concluding the episode with a reward estimate based on the output of the value network
image size	(84,84)	the size to which to rescale the original input into. This is done mainly because working on the original images is very expensive.
gamma	0.99	the discount factor

Table 2. Table 2: Forward convolution times [ms]. Notice that the MKL TF times are consistently smaller than the standard TF times. Data layout conversion times are not included in these measurements.

input size	kernel size	MKL TF		TF
		Phi	Xeon	Phi	Xeon
128,84,84,16	16,32,5,5	10.03	23.61	90.11	99.74
128,40,40,32	32,32,5,5	4.58	8.76	43.83	33.61
128,18,18,32	32,64,5,5	1.61	2.71	17.20	10.22
128,7,7,64	64,64,3,3	0.88	0.38	3.50	0.79

Table 3. Table 3: Backward data convolution times [ms]. TensorFlow times for the first layer are not listed since computing the gradient w.r.t the input of the model is unnecessary.

input size	kernel size	MKL TF		TF
		Phi	Xeon	Phi	Xeon
128,84,84,16	16,32,5,5	N/A	N/A	N/A	N/A
128,40,40,32	32,32,5,5	11.17	16.99	468.82	112.77
128,18,18,32	32,64,5,5	4.38	4.55	50.09	9.74
128,7,7,64	64,64,3,3	2.14	0.77	4.41	1.22

Table 4. Table 4: Backward filter convolution times [ms]. Please note very long time spent in the first layer by the standard TensorFlow convolution. It was possible to reduce it more than 10 times by using our implementation

input size	kernel size	MKL TF		TF
		Phi	Xeon	Phi	Xeon
128,84,84,16	16,32,5,5	8.97	29.63	1,236.98	368.18
128,40,40,32	32,32,5,5	6.33	19.55	343.73	114.72
128,18,18,32	32,64,5,5	2.52	6.07	36.74	28.82
128,7,7,64	64,64,3,3	2.31	3.18	7.38	5.57

Table 5. Table 5: Data layout conversion times [ms].

input size	kernel size	Forward		BWD Filter		BWD data
		Phi	Xeon	Phi	Xeon	Phi	Xeon
128,84,84,16	16,32,5,5	37.44	12.20	14.55	11.70	N/A	N/A
128,40,40,32	32,32,5,5	3.30	2.92	5.32	4.34	6.18	4.14
128,18,18,32	32,64,5,5	2.31	0.58	4.32	0.62	5.89	0.68
128,7,7,64	64,64,3,3	1.96	0.15	11.48	0.56	2.59	0.24

Table 6. Table 6: Mean scores and hyperparameters obtained for the best models for each game

game	learning rate	batch size	score mean	score max
Breakout-v0	0.00087	22	390.28	654
Pong-v0	0.00017	19	16.64	21
Riverraid-v0	0.00024	87	10,018.40	11570
Seaquest-v0	0.00160	162	1,823.41	1840
SpaceInvaders-v0	0.00032	14	764.70	2000

Full text

In this work we present our results on learning strategies in Atari games using a Convolutional Neural Network, the Math Kernel Library and TensorFlow 0.11rc0 machine learning framework. We also analyze effects of asynchronous computations on the convergence of reinforcement learning algorithms.

1 Introduction

In this work we approach the problem of learning strategies in Atari games from the hardware architecture perspective. We use a variation of the statistical model developed in [13, 14]. Using the provided code111https://github.com/deepsense-io/BA3C-CPU our experiments are easy to re-create and we encourage the reader to draw his own conclusions about how CPUs perform in the context of Atari games. Following [7, 13, 14] we treat Atari games as a key benchmark problem for modern reinforcement learning.

We use a statistical model consisting of approximately one million floating point numbers which are iteratively updated using a gradient descent algorithm described in [12].

At first glance a training of such model appears as a relatively straightforward task: a screen from the simulator is fed into the statistical model which decides which button must be pressed; over an episode of a game we estimate how the agent performs and calculate the loss accordingly and update the model so that the loss is reduced.

In practice filling details of the above scenario is quite challenging. In this work we accept a number of technical solutions presented in [13]. Our work also follows closely research done in [5], where a batch version of [13] is analyzed. We describe our algorithmic decisions in considerable detail in Section 2.2.

We obtained state-of-the-art results in all tested games (see Section 6) and in the process of obtaining them we detected certain interesting issues described in Sections 2.3, 6.2 related to batch sizes, learning rates and the asynchronous learning algorithm we use in this paper. The issues are illustrated by Figures 5 and 6. Apparently our algorithm relies on timely emptying of queues. If queues are growing, then updates are delayed and learning performance degenerates up to the point where the trained agent goes back to an essentially a random behavior. This in turn implies certain “preferred” sizes of batches as illustrated by Figure 8. Those batch sizes in turn imply “preferred” learning rates also visible in Figure 8.

Our contribution can be considered as a snapshot of the CPU performance in the domain of reinforcement learning illustrating engineering opportunities and obstacles one can encounter relying solely on a CPU hardware. We also contributed an integration of Google’s machine learning framework TensorFlow 0.11rc0 with Intel’s Math Kernel Library (MKL). Details of the integration are described in Section 5 and benchmarks comparing behavior of the out-of-the-box TensorFlow 0.11rc0 with our modified version are included in Section 2. Section 3 contains a description of our hardware. Let us underline that we relied on a completely standard Intel servers. Section 4 contains a brief characteristic of the MKL library and its currently available deep learning functionalities.

The learning times on our hardware described in Section 3 are very competitive (see Figure 7) and in a future work we are planning to bring it down to minutes using sufficiently large CPU clusters.

1.1 Related tools

This work would be impossible without a number of custom machine learning and reinforcement learning engineering tools. Our work is based on

• •

OpenAI Gym [7], an open source machine learning platform allowing a very easy access to a rich library of games, including Atari games,

• •

Google’s TensorFlow 0.11rc0, an open source machine learning framework [4] allowing for streamlined integration of various neural networks primitives (layers) implemented elsewhere,

• •

Tensorpack, an open source library [23] implementing a very efficient reinforcement learning algorithm,

• •

Intel’s Math Kernel Library 2017 (MKL) [19], a freely available library which implemented neural networks primitives (layers) and overall speeds up matrix and in particular deep learning computations on Intel’s processors.

1.2 Related work

Relation to [13].

Decisions in which we follow [13]. One of the key decisions is to run many independent agents in separate environments in an asynchronous way. In the training process in every environment we play an episode of 2000 steps (the number may be smaller if the agent dies). An input to the statistical model consists of 4 subsequent screens in the RGB format. An output of the statistical model is one of 18 possible moves of the controller. Over each episode the agent generates certain reward. The reward allows us to estimate how good were decisions made for every screen appearing during the episode. At every step an impact of the reward on decisions made earlier in the episode is discounted by a factor $\gamma$ ($0<\gamma\leq 1$).

Having computed rewards for a given episode we can update weights of the model according to rewards — this is done through gradient updates which are applied directly to the statistical model weights. The updates are scaled by a learning rate $\lambda$. Authors of [13] reported good CPU performance and this encouraged the experiment described in this paper.

Decisions left to readers of [13]. The key missing detail are all technical decisions related to communication between processes.

Relation to [5] and [18].

Since the publication of [14] a significant number of new results was obtained in the domain of Atari games, however to the best of our knowledge only the works [5] and [18] were focused on the hardware performance. In [5] authors modify the approach from [13] so it fits better into the GPU multi-core infrastructure. In this work we show that a similar modification can be also quite helpful for the CPU performance. This work can be considered a CPU variant of [5]. In [18] a significant speedup of the A3C algorithm was obtained using large CPU clusters. However, it is unclear if the method scales beyond the game of Pong. Also the announcement [18] does not contain neither technical details or implementation.

Relation to [22].

The fork of TensorFlow announced in [22] will offer a much deeper integration of TensorFlow and Intel’s Math Kernel Library (MKL). In particular it should resolve the dimensionality issue mentioned in Section 5.4. However, at the moment of writing of this paper we had to do the integration of these tools on our own, because the fork mentioned in [22] was not ready for our experiments.

Other references.

The work [14] approaches the problem of learning a strategy in Atari games through approximation of the $Q$-function, that is implicitly it learns a synthesized values of every move of a player in a given situation on the screen. We did not consider this method, because of overall weaker results and much longer training times comparing to the asynchronous methods in [13].

The DeepBench [8], the FALCON Library [3] and the study [1] compare a performance of CPU and GPU on neural network primitives (single convolutional and dense layers) as well as on a supervised classification problem. Our article can be considered a reinforcement learning variant of these works.

A recently published work [20] shows a very promising CPU-only results for agent training tasks. The learning algorithm proposed in [20] is a novel approach with yet untested stability properties. Our work focuses on a more established family of algorithms with better understood theoretical properties and applicability tested on a broader class of domains.

For a broad introduction to reinforcement learning we refer the reader to [21]. For a historical background on Atari games we refer to [14].

2 The Batch Asynchronous Advantage Actor Critic Algorithm (BA3C)

The Advantage Actor Critic algorithm (A2C) is a reinforcement learning algorithm combining positive aspects of both policy-based and value function based approaches to reinforcement learning. The results reported recently by Mnih et al. in [13] provide strong arguments for using its asynchronous version (A3C). After testing several implementations of this algorithm we found that a high quality open source implementation of this algorithm is provided in the TensorPack (TP) framework [23]. However, the differences between this variant, which resembles an algorithm introduced in [5], and the one described originally in [13] are significant enough to justify a new name. Therefore we will refer to this implementation as the Batch Asynchronous Advantage Actor Critic (BA3C) algorithm.222In [5] is proposed a different name GA3C derived from “hybrid CPU/GPU implementation of the A3C algorithm”. This seems a bit inconvenient, because it suggests a particular link between the batch algorithm and the GPU hardware; in this work we obtain good results for a similar algorithm running only on CPU.

2.1 Asynchronous reinforcement learning algorithms

Asynchronous reinforcement learning procedures are designed to use multiple concurrent environments to speed up the training process. This leaves an issue how the model or models are stored and synchronized between environments. We discuss some possible options in 2.2, including description of our own decisions.

Apart from obvious speedups resulting from utilizing concurrency, this approach has also some statistical consequences. Usually in one environment the subsequent states are highly correlated. This can have some adverse effects on the training process. However, when using multiple environments simultaneously, the states in each environment are likely to be significantly different, thus decorrelating the training points and enabling the algorithm to converge even faster.

2.2 BA3C – details of the implementation

The batch variant of the A3C algorithm was designed to better utilize massively parallel hardware by batching data points. Multiple environments are still used, but there’s only one instance of the model. This forces the extensive use of threading and message queues to decouple the part of the algorithm that generates the data from the one responsible for updates of the model. In a simple case of only one environment the BA3C algorithm consists of the steps described in algorithm 1.

When using multiple environments one can follow a similar approach - each environment could simply use the global model to predict the optimal action given its current state. Let us notice that the model always performs prediction on just a single data point from a single environment (i.e.: a single state vector of the environment). Obviously, this is far from optimal in terms of processing speed. Also accessing the shared model from different environments will quickly become a bottleneck. The two most popular approaches for solving this problem are:

• •

Maintaining several local copies of the model (one for each environment) and synchronizing them with a global model. This approach is used and extensively described in [13, 16, 17] and we refer to it as A3C.

• •

Using a single model and batching the predictions from multiple environments together (the “batch” variant, BA3C). This is much more suitable for use on massively parallel hardware [5].

The batch variant requires using the following queues for storing data:

Training queue

– stores the data points generated by the environments; the data points are used in training. See Figure 1.

Prediction requests queue

– stores the prediction requests made by the environments; the predictions are made according to the current weights stored in the model. See Figure 2.

Prediction results queue

– stores the results of the predictions made by the model; the predictions are later used by the environments for choosing actions. See Figure 3.

2.2.1 Hyperparameters

In Table 1 we list the most important hyperparameters of the algorithm is presented.

2.2.2 ConvNet architecture

We made rather minor changes to the original TensorPack ConvNet. The main focus of the changes was to better utilize the MKL convolution primitives to enhance the performance. The architecture is presented in the diagram below.

2.3 Effects of asynchronism on convergence

Training and prediction part of the above described algorithm work in separate threads and there’s a possibility that one of those parts will work faster than the other (in terms of data points processed per unit time). This is rarely an issue when the training thread is faster – in this case it’ll simply find out that the training queue is empty and wait for a batch of training data to be generated. This is inefficient since the hardware is not fully utilized when the train thread is waiting for data, but it should not impact the correctness of the algorithm.

A much more interesting case arises when data points are generated faster than can be consumed by the training thread. If we’re using default first-in-first-out training queue and this queue is not empty, then there’s some delay between the batch of data being generated by the prediction thread and it being used for training. It turns out that if this delay is large enough it will have detrimental effect on the convergence of the algorithm.

When there’s a significant delay between the generation of a batch and training on it, the training will be performed using a data point generated by an older model. That is because when the batch of data was “waiting” in the training queue, other batches were used for training and the model was updated. The number of such updates is equal to the size of the queue at the time when this batch was generated. Therefore the updates are performed using out-of-date training data which may have little to do with the current policy maintained by the current model.

Of course when this delay is small and the learning rate is moderate the current policy is almost equal to the “old” one used for generating the training batch and the training process will converge. In other cases one should have means of constraining the delay to force correct behavior.

The solution is to restrict the size of the training queue. This way, when the training thread is generating too many training batches it will at some point reach the full capacity of the queue and will be forced to wait until some batch is popped. Usually the the size of the training queue is set to ensure that the training can take place smoothly. What we found out, however, is that setting the queue capacity to extremely small values (i.e., less than five), has little if any impact on the overall training speed.

2.3.1 Impact of delay on convergence – experiments

This section describes a series of experiments we’ve carried out in order to establish how big a delay in the pipeline has to be to negatively impact the convergence. The setup involved inserting a fixed size first-in-first-out buffer between the prediction and training parts of the algorithm. This buffer’s task was to ensure a predefined delay in the algorithm was present. With this modification we were able to conduct a series of experiments for different sizes of this buffer (delays). The results are shown below.

Based on our results presented in the figures 5 and 6 we can conclude that even small delays have significant impact on the results and delays of more than 10 batches (1280 data points) effectively prevented the BA3C from converging. Therefore when designing an asynchronous RL algorithm it might be a good idea to try to streamline the pipeline as much as possible by making the queues as small as possible. This should not have significant effects on processing speed and can significantly improve obtained results.

3 Specification of involved hardware

3.1 Intel Xeon® (Broadwell)

We used Intel Xeon® E5 2600 v4 processors to perform benchmarks tests of convolutions. Xeon Broadwell is based on processor microarchitecture known as a “tick” [15] – a die shrink of an existing architecture, rather than a new architecture. In that sense, Broadwell is basically a Haswell made on Intel’s 14nm second generation tri-gate transistor process with few improvements to the micro-architecture. Important changes are: up to 22 cores per CPU; support for DDR4 memory up to 2400 MHz; faster floating point instruction performance; improved performance on large data sets. Results reported here are obtained on a system with two Intel Xeon® Processor E5 2689 (3.10 GHz, 10 core) with 128 GB of DDR4 2400MHz RAM, Intel Compute Module S2600TP and Intel Server Chassis H2312XXLR2. The system was running Ubuntu 16.04 LTS operating system. The code was compiled with GCC 5.4.0 and linked against the Intel MKL 2017 library (build date 20160802).

3.2 Intel Xeon® (Haswell)

Intel Xeon® E5 2600 v3 Processor, was used as base for series of experiments to test hyperparameters of our algorithm. Haswell brings, along with new microarchitecture, important features like AVX2. We used the Prometheus cluster with a peak performance of 2.4 PFlops located at the Academic Computer Center Cyfronet AGH as our testbed platform. Prometheus consists of more than 2,200 servers, accompanied by 279 TB RAM in total, and by two storage file systems of 10 PB total capacity and 180 GB/s access speed. Experiments were performed in single-node mode, each node consisting of two Intel Xeon®  E5-2680v3 processors with 24 cores at 2.5GHz with 128GB of RAM, with peak performance of 1.07 TFlops.

Xeon Haswell CPU allows effective computations of CNN algorithms, and convolutions in particular, by taking advantage of SIMD (single instruction, multiple data) instructions via vectorization and of multiple compute cores via threading. Vectorization is extremely important as these processors operate on vectors of data up to 256 bits long (8 single-precision numbers) and can perform up to two multiply and add (Fused Multiply Add, or FMA) operations per cycle. Processors support Intel Advanced Vector Extensions 2.0 (AVX2) vector-instruction sets which provide: (1) 256-bit floating-point arithmetic primitives, (2) Enhancements for flexible SIMD data movements. These architecture-specific advantages have been implemented in the Math Kernel Library (MKL) and used in deep learning framework Caffe [9], [2] resulting in improved convolutions performance.

4 The MKL library

The Intel Math Kernel Library (Intel MKL) 2017 introduces a set of Deep Neural Networks (DNN) [19] primitives for DNN applications optimized for the Intel architecture. The primitives implement forward and backward passes for the following operations: (1) Convolution: direct batched convolution, (2) Inner product, (3) Pooling: maximum, minimum, and average, (4) Normalization: local response normalization across channels and batch normalization, (5) Activation: rectified linear neuron activation (ReLU), (6) Data manipulation: multi-dimensional transposition (conversion), split, concatenation, sum, and scale. Intel MKL DNN primitives implement a plain C application programming interface (API) that can be used in the existing C/C**++** DNN frameworks, as well as in custom DNN applications.

5 Changes in TensorFlow 0.11rc0

5.1 Motivation

Preliminary benchmarks showed that the vast majority of computation time during training is spent performing convolutions. On CPU the single most expensive operation was the backward pass with respect to the convolution’s kernels, especially in the first layers working on the largest inputs. Therefore significant increases in performance had to be achieved by optimizing the convolution operation.

We considered the following approaches to this problem:

Tuning the current implementation of convolutions

– TensorFlow (TF) uses the Eigen [10] library as a backend for performing matrix operations on CPU. Therefore this approach would require performing changes in the code of this library. The matrix multiplication procedures used inside Eigen have multiple hyperparameters that determine the way in which the work is divided between the threads. Also, some rather strong assumptions about the configuration of the machine (e.g., its cache size) are made. This certainly leaves space for improvements, especially when optimizing for a very specific use-case and hardware.

Providing alternative implementation of convolutions

– The MKL library provides deep neural network operations optimized for the Intel architectures. Some tests of convolutions on a comparable hardware had already been performed by Baidu [8] and showed promising results. This also had the added benefit of leaving the original implementation unchanged thus making it possible for the user to decide which implementation (the default or the optimized one) to use.

We decided to employ the second approach that involved using the MKL convolution. A similar decision was taken also in the development of the Intel-focused fork of TensorFlow [22].

5.2 Implementation

TensorFlow provides a well documented mechanism for adding user-defined operations in C**++**, which makes it possible to load additional operations as shared objects. However, maintaining a build for a separate binary would make it harder to use some internal TF’s utilities and sharing code with the original convolution operation. Therefore we decided to fork the entire framework and provide the additional operations.

Another TF’s feature called ’labels’ made it very simple to provide several different implementations of the same operation in C**++** and choose between them from the python layer by specifying a ’label map’. This proved especially helpful while testing and benchmarking our implementation since we could quickly compare it to the original implementation.

The implementation consisted of linking against the MKL library and providing the three additional operations: (1) MKL convolution forward pass, (2) MKL convolution backpropagation w.r.t. the input feature map, (3) MKL convolution backpropagation w.r.t. the kernels.

The code of these operations formed a glue layer between the TF’s and MKL’s programming interfaces. The computations were performed inside highly optimized MKL primitives.

5.3 Benchmark results

Multiple benchmarks were conducted in order to assess the performance of our implementation. They are focused on a specific 4-layer ConvNet architecture used for processing the Atari input images. The results are shown below.

Tables 2, 3 and 4 show the benchmark results for the TensorFlow modified to use MKL and standard TensorFlow. Measurements consist of the times of performing convolutions with specific parameters (input and filter sizes) for Xeon® and Xeon Phi® CPUs. The same convolution parameters were used in the convolutional network used in the atari games experiments.

The results show that the MKL convolutions can be substantially faster than the ones implemented in TensorFlow. For some operations a speed-up of more than 10 times was achieved. The results agree with the ones reported in [8]. It is also worth noticing that most of the time is spent in the first layer which is responsible for processing the largest images.

5.4 Possible improvements

The data layout can have a tremendous impact on performance of low-level array operations. In turn, efficiency of these operations is critical for performance of higher-level machine learning algorithms.

TensorFlow and MKL have radically different philosophies of storing visual data. TensorFlow uses mostly its default “NHWC” format, in which pixels with the same spatial location but different channel indices are placed close to each other in memory. Some operations also provide the “NCHW” format widely used by other deep learning frameworks such as Caffe [11]. On the other hand MKL does not have a predefined default format, rather it is designed to easily connect MKL layers to one another. In particular, the same operation can require different data layouts depending on the sizes of its input (e.g. the number of input channels). This is supposed to ensure that the number of intermediate “conversions” or “transpositions” in the pipeline is minimal, while at the same time letting each operation use its preferred data layout.

It is important to note that our implementation provided an alternative “MKL” implementation only for the convolution. We did not provide similar alternatives for max pooling, ReLU etc. This forced us to repeatedly convert the data between the TF’s NHWC format and the formats required by the MKL convolution. Obviously this is not an optimal approach, however, implementing it optimally would most probably require significant changes in the very heart of the framework – its compiler. This task was beyond the scope of the project, but it’s certainly feasible and with enough effort our implementation’s performance could be even further improved. The times necessary to perform data conversions are provided in the Table 5.

6 Results

6.1 Game scores and overall training time

By using the custom convolution primitives from the MKL library it was possible to increase the training speed by a factor of 3.8 (from 151.04 examples/s to 517.12 examples/s). This made it possible to train well performing agents in under 24 hours. As a result, novel concepts and improvements to the algorithm can now be tested more quickly, possibly leading to further advances in the field of reinforcement learning. The increase in speed was achieved without hurting the results obtained by the agents trained. Example training curves for 3 different games are presented in the Figure 7.

6.2 Batch size and learning rate tuning

Using the previously described pipeline optimized for better CPU performance we conducted a series of experiments designed to determine the optimal batch size and learning rate hyperparameters. The experiments were performed using the random search method [6]. For each hyperparameter its value was drawn from a loguniform distribution defined on a range $[10^{-4},10^{-2}]$ for learning rate and $[2^{1},2^{10}]$ for batch size. Overall, over 200 experiments were conducted in this manner for 5 different games. The results are presented in the figures 8,9 below. It appears that for the 5 games tested one could choose a combination of learning rate and batch size that would work reasonably well for all of them. However, the optimal settings for specific games seem to diverge.

As one could expect when using large batch sizes, better results were obtained with greater learning rate’s. This is most probably caused by the stabilizing effects of bigger batch sizes on the mean gradient vector used for training. For smaller batch sizes using the same learning rate would cause instabilities, impeding the training process.

Overall, batch size of around 32 a learning rate of the order of $10^{-4}$ seems to have been a general good choice for the games tested. The detailed listing of the best results obtained for each game is presented in the Table 6.

7 Conclusions and further work

Preliminary results contained in this work can be considered as a next step in reducing the gap between CPU and GPU performance in deep learning applications. As shown in this paper, in the area of reinforcement learning and in the context of asynchronous algorithms, CPU-only algorithms already achieve a very competitive performance.

As the most interesting future research direction we perceive extending results of [18] and tuning of performance of asynchronous reinforcement learning algorithms on large computer clusters with the idea of bringing the training time down from hours to minutes.

Constructing a compelling experiment for the Xeon Phi® platform also seems to be an interesting challenge. Our current approach would require a significant modification because of much slower single core performance of Xeon Phi®. However, preliminary results on the Pong game are quite promising with a state-of-the-art results obtained in 12 hours on a single Xeon Phi® server.