An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents

TL;DR

This paper introduces the Atari Zoo framework, a collection of trained deep reinforcement learning models for Atari games, enabling easier analysis, visualization, and comparison of different algorithms' learned representations.

Contribution

It provides a scalable, accessible platform with trained models and analysis tools, facilitating understanding and comparison of deep RL algorithms on Atari benchmarks.

Findings

Initial comparisons reveal distinct performance patterns among algorithms.

Visualization uncovers differences in learned representations.

The framework simplifies analysis of deep RL models.

Abstract

Much human and computational effort has aimed to improve how deep reinforcement learning algorithms perform on benchmarks such as the Atari Learning Environment. Comparatively less effort has focused on understanding what has been learned by such methods, and investigating and comparing the representations learned by different families of reinforcement learning (RL) algorithms. Sources of friction include the onerous computational requirements, and general logistical and architectural complications for running Deep RL algorithms at scale. We lessen this friction, by (1) training several algorithms at scale and releasing trained models, (2) integrating with a previous Deep RL model release, and (3) releasing code that makes it easy for anyone to load, visualize, and analyze such models. This paper introduces the Atari Zoo framework, which contains models trained across benchmark Atari games, in an easy-to-use format, as well as code that implements common modes of analysis and connects such models to a popular neural network visualization library. Further, to demonstrate the potential of this dataset and software package, we show initial quantitative and qualitative comparisons between the performance and representations of several deep RL algorithms, highlighting interesting and previously unknown distinctions between them.

Tables

Table 1. Table S1 : Average F1 scores by game. The score is an unweighted average of F1 scores across all algorithm. Lower scores indicate games for which different DRL algorithms are less distinguishable from each other.

Game	Mean F1
Amidar	0.96
Assault	0.86
Asterix	0.96
Asteroids	0.96
Atlantis	0.73
Enduro	0.92
Frostbite	0.94
Gravitar	0.92
Kangaroo	0.93
Seaquest	0.88
Skiing	0.9
Venture	0.96
Zaxxon	0.96

Table 2. Table S2 : A2C Hyperparameters. Population sizes are incremented to account for elites ( + 1 1 +1 ). Many of the unusual numbers were found via preliminary hyperparameter searches in other domains.

Hyperparameter	Setting
Learning Rate	7e-5
$\tau$	1.0
Value Function Loss Coefficient	0.5
Entropy Loss Coefficient	0.01
Discount factor	0.99

Table 3. Table S3 : Ape-X Hyperparameters. For more details on what these parameters signify, see \citepNew horgan:distributed.

Hyperparameter	Setting
Buffer Size	$2^{21}$
Number of Actors	384
Batch Size	512
n-step	3
gamma	0.99
gradient clipping	40
target network period	2500
Prioritized replay $(\alpha ,\beta )$	(0.6, 0.4)
Adam Learning rate	0.00025 / 4

Table 4. Table S4 : GA Hyperparameters. For more details on what these parameters signify, see \citepNew such:deep.

Hyperparameter	Setting
$\sigma$ (Mutation Power)	0.002
Population Size	1000
Truncation Size	20

Table 5. Table S5 : ES Hyperparameters. For more details on what these parameters signify, see \citepNew salimans:es, conti:improving.

Hyperparameter	Setting
$\sigma$ (Mutation Power)	0.02
Virtual Batch Size	128
Population Size	5000
Learning Rate	0.01
Optimizer	Adam
L2 Regularization Coefficient	0.005

Table 6. Table S6 : IMPALA Hyperparameters. For more details on what these parameters signify, see \citepNew espeholt:impala.

Hyperparameter	Setting
Number of actors	25
Image Width	84
Image Height	84
Grayscaling	Yes
Action Repetitions	4
Max-pool over last N action repeat frames	2
Frame Stacking	4
End of episode when life lost	Yes
Reward Clipping	[-1, 1]
Unroll Length (n)	20
Batch size	32
Discount ($\gamma$)	0.99
Baseline loss scaling	0.5
Entropy Regularizer	0.01
RMSProp momentum	0.0
RMSProp $ϵ$	0.01
Learning rate	0.0006
Clip global gradient norm	40.0
Learning rate schedule	Anneal linearly to 0 from beginning to end of training

Full text

1 Introduction

Since its introduction the Atari Learning Environment (ALE; (Bellemare et al., 2013)) has been an important reinforcement learning (RL) testbed. It enables easily evaluating algorithms on over 50 emulated Atari games spanning diverse gameplay styles, providing a window on such algorithms’ generality. Indeed, surprisingly strong results in ALE with deep neural networks (DNNs), published in Nature (Mnih et al., 2015), greatly contributed to the current popularity of deep reinforcement learning (DRL).

Like other machine learning benchmarks, much effort aims to quantitatively improve state-of-the-art (SOTA) scores. As the DRL community grows, a paper pushing SOTA is likely to attract significant interest and accumulate citations. While improving performance is important, it is equally important to understand what DRL algorithms learn, how they process and represent information, and what are their properties, strengths, and weaknesses. These questions cannot be answered through simple quantitative measurements of performance across the ALE suite of games.

Compared to pushing SOTA, much less work has focused on understanding, interpreting, and visualizing products of DRL; in particular, little research compares DRL algorithms across dimensions other than performance. This paper thus aims to alleviate the considerable friction for those looking to rigorously understand the qualitative behavior of DRL agents. Three main sources of such friction are: (1) the significant computational resources required to run DRL at scale, (2) the logistical tedium of plumbing the products of different DRL algorithms into a common interface, and (3) the wasted effort in re-implementing standard analysis pipelines (like t-SNE embeddings of the state space (Mnih et al., 2015), or activation maximization for visualizing what neurons in a model represent (Erhan et al., 2009; Olah et al., 2018; Nguyen et al., 2017; Simonyan et al., 2013; Yosinski et al., 2015; Mahendran and Vedaldi, 2016)). To address these frictions, this paper introduces the Atari Zoo, a release of trained models spanning major families of DRL algorithms, and an accompanying open-source software package111https://github.com/uber-research/atari-model-zoo that enables their easy analysis, comparison, and visualization (and similar analysis of future models). In particular, this package enables easily downloading particular frozen models of interest from the zoo on-demand, further evaluating them in their training environment or modified environments, generating visualizations of their neural activity, exploring compressed visual representations of their behavior, and creating synthetic input patterns that reveal what particular neurons most respond to.

To demonstrate the promise of this model zoo and software, this paper presents an initial analysis of the products of seven DRL algorithms spanning policy gradient, value-based, and evolutionary methods222While evolutionary algorithms are excluded from some definitions of RL, their inclusion in the zoo can help investigate what distinguishes such black-box optimization from more traditional RL.: A2C (policy-gradient; (Mnih et al., 2016)), IMPALA (policy-gradient; (Espeholt et al., 2018)), DQN (value-based; (Mnih et al., 2015)), Rainbow (value-based; (Hessel et al., 2017)), Ape-X (value-based; (Horgan et al., 2018)), ES (evolutionary; (Salimans et al., 2017)), and Deep GA (evolutionary; (Such et al., 2017)). The analysis illuminates differences in learned policies across methods that are independent of raw score performance, highlighting the benefit of going beyond simple quantitative measures and of having a unifying software framework that enables analyses with multiple, different, complementary techniques and applying them across many RL algorithms.

2 Background

2.1 Visualizing Deep Networks

One line of DNN research focuses on visualizing the internal dynamics of a DNN (Yosinski et al., 2015) or examines what particular neurons detect or respond to (Erhan et al., 2009; Zeiler and Fergus, 2014; Olah et al., 2018; Nguyen et al., 2017; Simonyan et al., 2013; Yosinski et al., 2015; Mahendran and Vedaldi, 2016). The hope is to gain more insight into how DNNs are representing information, motivated both to enable more interpretable decisions from these models (Olah et al., 2018) and to illuminate previously unknown properties about DNNs (Yosinski et al., 2015). For example, through live visualization of all activations in a vision network responding to different images, Yosinski et al. (2015) highlighted that representations were often surprisingly local (as opposed to distributed), e.g. one convolutional filter proved to be a reliable face detector. One practical value of such insights is that they can catalyze future research. The Atari Zoo enables animations in the spirit of Yosinski et al. (2015) that show an agent’s activations as it interacts with a game, and also enables creating synthetic inputs via activation maximization (Erhan et al., 2009; Zeiler and Fergus, 2014; Olah et al., 2018; Nguyen et al., 2017; Simonyan et al., 2013; Yosinski et al., 2015; Mahendran and Vedaldi, 2016), specifically by connecting DRL agents to the Lucid visualization package (Luc, 2018).

2.2 Understanding Deep RL

While much more visualization and understanding work has been done for vision models than for DRL, a few papers directly focus on understanding DRL agents (Greydanus et al., 2017; Zahavy et al., 2016), and many others feature some analysis of DRL agent behavior (often in the form of t-SNE diagrams of the state space; see (Mnih et al., 2015)). One approach to understanding DRL agents is to investigate the learned features of models (Greydanus et al., 2017; Zahavy et al., 2016). For example, Zahavy et al. (2016) visualize what pixels are most important to an agent’s decision by using gradients of decisions with respect to pixels. Another approach is to modify the DNN architecture or training procedure such that a trained model will have more interpretable features (Annasamy and Sycara, 2018). For example, Annasamy and Sycara (2018) augment a model with an attention mechanism and a reconstruction loss, hoping to produce interpretable explanations as a result.

The software package released here is in the spirit of the first paradigm. It facilitates understanding the most commonly applied architectures instead of changing them, although it is designed also to accommodate importing in new vision-based DRL models, and could thus be also used to analyze agents explicitly engineered to be interpretable. In particular, the package enables re-exploring many past DRL analysis techniques at scale, and across algorithms, which were previously applied only for one algorithm and across only a handful of hand-selected games.

2.3 Model Zoos

A useful mechanism for reducing friction for analyzing and building upon models is the idea of a model zoo, i.e. a repository of pre-trained models that can easily be further investigated, fine-tuned, and/or compared (e.g. by looking at how their high-level representations differ). For example, the Caffe website includes a model zoo with many popular vision models, as do Tensorflow, Keras, and PyTorch. The idea is that training large-scale vision networks (e.g. on the ImageNet dataset) can take weeks with powerful GPUs, and that there is little reason to constantly reduplicate the effort of training. Pre-trained word-embedding models are often released with similar motivation, e.g. for Word2Vec or GLoVE. However, such a practice is much less common in the space of DRL; one reason is that so far, unlike with vision models and word-embedding models, there are few other down-stream tasks from which Atari DRL agents provide obvious value. But, if the goal is to better understand these models and algorithm, both to improve them and to use them safely, then there is value in their release.

The recent Dopamine reproducible DRL package (Bellemare et al., 2018) released trained ALE models; it includes final checkpoints of models trained by several DQN variants. However, in general it is non-trivial to extract TensorFlow models from their original context for visualization purposes, and to compare agent behavior across DRL algorithms in the same software framework (e.g. due to slight differences in image preprocessing), or to explore dynamics that take place over learning, i.e. from intermediate checkpoints. To remedy this, for this paper’s accompanying software release, the Dopamine checkpoints were distilled into frozen models that can be easily loaded into the Atari Zoo framework; and for algorithms trained specifically for the Atari Zoo, we distill intermediate checkpoints in addition to final ones.

3 Generating the Zoo

The approach is to run several validated implementations of DRL algorithms and to collect and standardize the models and results, such that they can then be easily used for downstream analysis and synthesis. There are many algorithms, implementations of them, and different ways that they could be run (e.g. different hyperparameters, architectures, input representations, stopping criteria, etc.). These choices influence the kind of post-hoc analysis that is possible. For example, Rainbow most often outperforms DQN, and if only final models are released, it is impossible to explore scientific questions where it is important to control for performance.

We thus adopted the high level principles that the Atari Zoo should hold as many elements of architecture and experimental design constant across algorithms (e.g. DNN structure, input representation), should enable as many types of downstream analysis as possible (e.g. by releasing checkpoints across training time), and should make reasonable allowances for the particularities of each algorithm (e.g. ensuring hyperparameters are well-fit to the algorithm, and allowing for differences in how policies are encoded or sampled from). The next paragraphs describe specific design choices.

3.1 Frozen Model Selection Criteria

To enable the platform to facilitate a variety of explorations, we release multiple frozen models for each run, according to different criteria that may be useful to control for when comparing learned policies. The idea is that depending on the desired analysis, controlling for samples, or for wall-clock, or for performance (i.e. comparing similarly-performing policies) will be more appropriate. In particular, in addition to releasing the final model for each run, additional are models taken over training time (at one, two, four, six, and ten hours); over game frame samples (400 million, and 1 billion frames); over scores (if an algorithm reaches human level performance); and also a model before any training, to enable analysis of how weights change from their random initialization. The hope is that these frozen models will cover a wide spectrum of possible use cases.

3.2 Algorithm Choice

One important choice for the Atari Zoo is which DRL algorithms to run and include. The main families of DRL algorithms that have been applied to the ALE are policy gradients methods like A2C (Mnih et al., 2016), value-based methods like DQN (Mnih et al., 2015), and black-box optimization methods like ES (Salimans et al., 2017) and Deep GA (Such et al., 2017). Based on representativeness and available trusted implementations, the particular algorithms chosen to train included two policy gradients algorithms (A2C (Mnih et al., 2016) and IMPALA (Espeholt et al., 2018)), two evolutionary algorithms (ES (Salimans et al., 2017) and Deep GA (Such et al., 2017)), and one value-function based algorithm (a high-performing DQN variant, Ape-X; (Horgan et al., 2018)). Additionally, models are also imported from the Dopamine release (Bellemare et al., 2018), which include DQN (Mnih et al., 2015) and a sophisticated variant of it called Rainbow (Hessel et al., 2017). Note that from the Dopamine models, only final models are currently available. Hyperparameters and training details for all algorithms are available in supplemental material section S3. We hope to include models from additional algorithms in future releases.

3.3 DNN Architecture and Input Representation

All algorithms are run with the DNN architecture from Mnih et al. (2015), which consists of three convolutional layers (with filter size 8x8, 4x4, and 3x3, followed by a fully-connected layer). For most of the explored algorithms, the fully-connected layer connects to an output layer with one neuron per valid action in the underlying Atari game. However, A2C and IMPALA have an additional output that approximates the state value function; Ape-X’s architecture features dueling DQN (Wang et al., 2015), which has two separate fully-connected streams; and Rainbow’s architecture includes C51 (Bellemare et al., 2017), which uses many outputs to approximate the distribution of expected Q-values.

Atari frames are 210x160 color images (see figure 1a); the canonical DRL representation is a a tensor consisting of the four most recent observation frames, grayscaled and downsampled to 84x84 (figure 1b). By including some previous frames, the aim is to make the game more fully-observable, to boost performance of the feed-forward architectures that are currently most common in Atari research (although recurrent architectures offer possible improvements (Mnih et al., 2016; Espeholt et al., 2018)). One useful Atari representation that is applied in post-training analysis in this paper, is the Atari RAM state, which is only 1024 bits long but encompasses the true underlying state (figure 1c).

3.4 Data Collection

All algorithms are run across 55 Atari games, for at least three independent random weight initializations. Regular checkpoints were taken during training; after training, the checkpoints that best fit each of the desired criteria (e.g. 400 million frames or human-level performance) were frozen and included in the zoo. The advantage of this post-hoc culling is that additional criteria can be added in the future, e.g. if Atari Zoo users introduce a new use case, because the original checkpoints are archived. Log files were stored and converted into a common format that are also released with the models, to aid future performance curve comparisons for other researchers. Each frozen model was run post-hoc in ALE for $2500$ timesteps to generate cached behavior of policies in their training environment, which includes the raw game frames, the processed four-frame observations, RAM states, and high-level representations (e.g. neural representations at hidden layers). As a result, it is possible to do meaningful analysis without ever running the models themselves.

4 Quantitative Analysis

The open-source software package released with the acceptance of this work provides an interface to the Atari Zoo dataset, and implements several common modes of analysis. Models can be downloaded with a single line of code; and other single-line invocations interface directly with ALE and return the behavioral outcome of executing a model’s policy, or create movies of agents superimposed with neural activation, or access convolutional weight tensors. In this section, we demonstrate analyses the Atari Zoo software can facilitate, and highlight some of its built-in features. For many of the analyses below, for computational simplicity we study results in a representative subset of 13 ALE games used by prior research (Such et al., 2017), which we refer to here as the analysis subset of games.

4.1 Convolutional Filter Analysis

While understanding a DNN only by examining its weights is challenging, weights directly connected to the input can often be interpreted. For example, from visualizing the weights of the first convolutional layer in a vision model, Gabor-like edge detection filters are nearly always present. An interesting question is if Gabor-like filters also arise when DRL algorithms are trained from pixel input (as is done here). In visualizing filters across games and DRL algorithms, edge-detector-like features sometimes arise in the gradient-based methods, but they are seemingly never as crisp as in vision models; this may because ALE lacks the visual complexity of natural images. In contrast, the filters in the evolutionary models are less regular. Representative examples across games and algorithms are shown in supplemental figure S1.

Learned filters commonly are tiled similarly across time (i.e. across the four DNN input frames), with past frames having lower-intensity weights. One explanation is that reward gradients are more strongly influenced by present observations. To explore this systematically, across games and algorithms we examined the absolute magnitude of filter weights connected to the present frame versus the past. In contrast to the gradient-based methods the evolutionary methods show no discernable preference across time (supplemental figure S2), again suggesting that their learning differs qualitatively from the gradient-based methods. Interestingly, a rigorous information-theoretic approximation of memory usage is explored by Dann et al. (2016) in the context of DQN; our measure well-correlates with theirs despite the relative simplicity of exploring only filter weight strength (supplemental section S1.1).

4.2 Robustness to Observation Noise

An important property is how agents perform in slightly out-of-distribution (OOD) situations; ideally they would not catastrophically fail in the face of nominal change. While it is difficult to freely alter the ALE game dynamics (without learning how to program in 6502 assembly code), it is possible to systematically distort observations. Here we explore one simple OOD change to observations by adding increasingly severe noise to the observations input to DNN-based agents, and observe how their evaluated game score degrades. The motivation is to discover whether some learning algorithms are learning more robust policies than others. The results show that with some caveats, methods with a direct representation of the policy appear more robust to observation noise (supplemental figure S4). A similar study conducted for robustness to parameter noise (supplemental section S1.2) tentatively suggests that actor-critic methods are more robust to such noise.

4.3 Distinctiveness of Learned Policies

To explore the distinctive signature of solutions discovered by different DRL algorithms, we train image classifiers to identify the generating DRL algorithm given states sampled from independent runs of each algorithm (details are in supplemental section S1.3). Supplemental figure S6 shows the confusion matrix for Seaquest, wherein a cluster of policy search methods (A2C, ES, and GA) have the most inter-class confusion, reflecting (as confirmed in later sections) that these algorithms tend to converge to the same sub-optimal behavior in this game; results are qualitatively similar when tabulated across the analysis subset of games (supplemental figure S7).

5 Visualization

We next highlight the Atari Zoo’s capabilities to quickly and systematically visualize policies, which broadly can be divided into three categories: Direct policy visualization, dimensionality reduction, and neuron activation maximization.

5.1 Animations to Inspect Policy and Activations

To quickly survey the solutions being learned, our software generates grids of videos, where one grid axis spans different DRL algorithms, and the other axis covers independent runs of the algorithm. Such videos can highlight when different algorithms are converging to the same local optimum (e.g. supplemental figure S9 shows a situation where this is the case for A2C, ES, and the GA; video: http://bit.ly/2XpD5kO).

To enable investigating the internal workings of the DNN, our software generates movies that display activations of all neurons alongside animated frames of the agent acting in game. This approach is inspired by the deep visualization toolbox (Yosinski et al., 2015), but put into a DRL context. Supplemental figure S10 shows how this tool can lead to recognizing the functionality of particular high-level features (video: http://bit.ly/2tFHiCU); in particular, it helped to identify a submarine detecting neuron on the third convolution layer of an Ape-X agent. Note that for ES and GA, no such specialized neuron was found; activations seemed qualitatively more distributed for those methods.

5.2 Image Patches that Maximally Excite Filters

One automated technique for uncovering the functionality of a particular convolutional filter is to find which image patches evoke from it the highest magnitude activations. Given a trained DRL agent and a target convolution filter to analyze, observations from the agent interacting with its ALE training environment are input to the agent’s DNN, and resulting maps of activations from the filter of interest are stored. These maps are sorted by the single maximum activation within them, and the geometric location within the map of that maximum activation is recorded. Then, for each of these top-most activations, the specific image patch from the observation that generated it is identified and displayed, by taking the receptive field of the filter into account (i.e. modulated by both the stride and size of the convolutional layers). As a sanity check, we validate that the neuron identified in the previous section does indeed maximally fire for submarines (figure 2).

5.3 Dimensionality Reduction

Dimensionality reduction provides another view on agent behavior; often DRL research includes t-SNE plots of agent DNN representations that summarize behavior in the domain (Mnih et al., 2015). Our software includes such an implementation (supplemental figure S12).

However, such an approach relies on embedding the high-level representation of one agent; it is unclear how to apply it to create an embedding appropriate for comparisons of different independent runs of the same algorithm, or runs from different DRL algorithms. As an initial approach, we implement an embedding based on the Atari RAM representation (which is the same across algorithms and runs, but distinct between games). Like the grid view of agent behaviors and the state-distinguishing classifier, this t-SNE tool provides high-level information from which to compare runs of or between different algorithms (figure 3); details of this approach are provided in supplemental section S2.1.

5.4 Synthesizing Inputs to Understand Neurons

While the previous sections explore DNN activations in the context of an agent’s training environment, another approach is to optimize synthetic input images that stimulate particular DNN neurons. Variations on this approach have yielded striking results in vision models (Nguyen et al., 2017; Olah et al., 2018; Simonyan et al., 2013); the hope is that these techniques could yield an additional view on DRL agents’ neural representations. To enable this analysis, we leverage the Lucid visualization library (Luc, 2018); in particular, we create wrapper classes that enable easy integration of Atari Zoo models into Lucid, and release Jupyter notebooks that generate synthetic inputs for different DRL models.

We now present a series of synthetic inputs generated by the Lucid library across a handful of games that highlight the potential of these kinds of visualizations for DRL understanding (further details of the technique used are described in supplemental section S2.2. We first explore the kinds of features learned across depth. Supplemental figure S13 supports what was learned by visualizing the first-layer filter weights for value-based networks (section 4.1; i.e. showing that first convolution layers in the value-based networks appear to be learning edge-detector features). The activation videos of section 5.1 and the patch-based approach of section 5.2 help to provide grounding, showing that in the context of the game, some first-layer filters detect the edges of the screen, in effect to serve as location anchors, while others encode concepts like blinking objects (see figure S11). Supplemental figure S14 explores visualizing later-layer convolution filters, and figure 4 show inputs synthesized to maximize output neurons, which sometimes yields interpretable features.

Such visualizations can also reveal that critical features are being attended to (figure 5 and supplemental figure S15). Overall, these visualizations demonstrate the potential of this kind of technique, and we believe that many useful further insights may result from a more systematic application and investigation of this and many of the other interesting visualization techniques implemented by Lucid, which can now easily be applied to Atari Zoo models. Also promising would be to further explore regularization to constrain the space of synthetic inputs, e.g. a generative model of Atari frames in the spirit of Nguyen et al. (2017) or similar works.

6 Discussion and Conclusions

There are many follow-up extensions that the initial explorations of the zoo raise. One natural extension is to include more DRL algorithms (e.g. TRPO or PPO (Schulman et al., 2017)). Beyond algorithms, there are many alternate architectures that might have interesting effects on representation and decision-making, for example recurrent architectures, or architectures that exploit attention. Also intriguing is examining the effect of the incentive driving search: Do auxiliary or substitute objectives qualitatively change DRL representations, e.g. as in UNREAL (Jaderberg et al., 2016), curiosity-driven exploration (Pathak et al., 2017), or novelty search (Conti et al., 2017)? How do the representations and features of meta-learning agents such as MAML (Finn et al., 2017) change as they learn a new task? Finally, there are other analysis tools that could be implemented, which might illuminate other interesting properties of DRL algorithms and learned representation, e.g. the image perturbation analysis of Greydanus et al. (2017) or a variety of sophisticated neuron visualization techniques (Nguyen et al., 2017). We welcome community contributions for these algorithms, models, architectures, incentives, and tools.

While the main motivation for the zoo was to reduce friction for research into understanding and visualizing the behavior of DRL agents, it can also serve as a platform for other research questions. For example, having a zoo of agents trained on individual games, for different amounts of data, also would reduce friction for exploring transfer learning within Atari, i.e. whether experience learned on one game can quickly benefit on another game. Also, by providing a huge library of cached rollouts for agents across algorithms, the zoo may be interesting in the context of learning from demonstrations, or for creating generative models of games. In conclusion, we look forward to seeing how this dataset will be used by the community at large.

Supplementary Material

The following sections contain additional figures, and describe in more detail the experimental setups applied in the paper’s experiments.

S1 Quantitative Analysis Details

Figure S1 shows a sampling of first-layer convolutional filters from final trained models, and figure S2 highlights that such filters often differentially attend to the present over the past.

S1.1 Further Study of Temporal Bias in DQN

As an exploration of the connection between the information theoretic measure of memory-dependent action in \citetNewdann2016memory and the pattern highlighted in this paper (i.e. the strength of filter weights in the first layer of convolutions may highlight a network’s reliance on the past), we examined first-layer filters in DQN across all 55 games. A simple metric of present-focus is the ratio of average weight magnitudes for the past three frames to the present frame. When sorted by this metric (see figure S3), there is high agreement with the 8 games identified by \citetNewdann2016memory that use memory. In particular, three out of the top four games identified by our metric align with theirs; as do six out of the top twelve games, considered among the games that overlap between their 49 and our 55.

S1.2 Observation and Parameter Noise Details

Figure S4 shows robustness to observation noise for games in the analysis subset. Beyond observation noise, another interesting property of learning algorithms is the kind of local optimum they find in the parameter space, i.e. whether the learned function is smooth or not in the area of a found solution. One gross tool for examining this property is to test the robustness of policies to parameter perturbations. It is plausible that the evolutionary methods would be more robust in this way, given that they are trained through parameter perturbations. To measure this, we perturb the convolutional weights of each DRL agent with increasingly severe normally-distributed noise. We perturbed the convolutional weights only, because that part of the DNN is identical across agents, whereas the fully-connected layers sometimes vary in structure across DRL algorithms (e.g. value-based algorithms like Rainbow or Ape-X that include features that require post-convolutional architectural changes). Figure S5 shows the results across games and algorithms; while no incontrovertible trend exists, the two policy-gradient methods (A2C and Impala) show greater relative robustness than the other methods.

Note that for both algorithm-best performance plots (i.e. figures S4a and S5a), three games were excluded from analysis because at least one of the DRL algorithms performed worse than a random policy before pertubation; including them would have conflated performance and robustness, which would undermine the purpose of the plot.

S1.3 Distinctiveness of Policies Learned by Algorithms

We use only the “present” channel of each gray-scale observation frame (i.e. without the complete four-frame stack) to train a classifier for each game. The classifier consists of two convolution layers and two fully connected layers, and is trained with early stopping to avoid overfitting. For each game, 2501 frames are collected from multiple evaluations by each model. The reported classification results use 20% of the frames as test set. Figure S6 visualizes the confusion matrix for Seaquest frame classification, while figure S7 shows a confusion matrix summed across games.

We also provide summaries of F1 classification performance: Figure S8 summarizes classification performance across DRL algorithms and games, while table S1 shows performance averaged across games.

S2 Visualization Details

This section provides more details and figures for the visualization portion of the paper’s analysis. Figure S9 shows one frame of a collage of simultaneous videos that give a quick high-level comparison of how different algorithms and runs are solving an ALE environment. Figure S9 shows one frame of a video that simultaneously shows a DNN agent acting in an ALE environment and all of the activations of its DNN.

Figure S11 shows a second example of how the image-patch finder can help ground out what particular DNN neurons are learning.

S2.1 t-SNE Details

To visualize RAM states and high-level DNN representations in 2D, as is typical in t-SNE analysis PCA is first applied to reduce the number of dimensions (to 50), followed by 3000 t-SNE iterations with perplexity of 30. The dimensionality reduction of RAM states is applied across all available runs of DRL algorithms to be jointly embedded. In contrast, dimensionality reduction of high-level DNN representations is particular to a specific model trained by a single DRL algorithm (i.e. each run of a DRL algorithm learns its own distinct representation).

S2.2 Synthetic Input Generation Details

We use the lucid library \citepNewLucid to visualize what types of inputs maximize neuron activations throughout the agents’ networks. This study used the trained checkpoints provided by Dopamine \citepNewbellemare:dopamine for DQN and Rainbow (although it could be applied to any of the DRL algorithms in the Atari Zoo). These frozen graphs are then loaded as part of a Lucid model and an optimization objective is created.

An input pattern to the network (consisting of a stack of four 84x84 pixel screens) is optimized to maximize the activations of the desired neurons. Initially, the four 84x84 frames are initialized with random noise. The result of optimization ideally yields visualizations that reveal qualitatively what features the neurons have learned to capture. As recommended in \citetNewolah2017feature and \citetNewmahendran2016visualizing we apply regularization to produce clearer results; for most images we use only image jitter (i.e. randomly offsetting the input image by a few pixels to encourage local translation invariance). For some images, we found it helpful to add total variation regularization (to encourage local smoothness; see \citetNewmahendran2016visualizing) and L1 regularization (to encourage pixels that are not contributing to the objective to become zero) on the optimized image.

S3 DRL Algorithm Details and Hyperparameters

This section describes the implementations and hyperparameters used for training the models released with the zoo. The DQN and Rainbow models come from the Dopamine model release \citepNewbellemare:dopamine111The hyperparameters and training details for Dopamine can be found in https://github.com/google/dopamine/tree/master/baselines/. The following sections describe the algorithms for the newly-trained models released with this paper.

S3.1 A2C

The implementation of A2C \citepNewmnih2016asynchronous that generated the models in this paper was derived from the OpenAI baselines software package \citepNewdhariwal:baselines. It ran with 20 parallel worker threads for 400 million frames; checkpoints occurred every 4 million frames. Hyperparameters are listed in table S2.

S3.2 Ape-X

The implementation of Ape-X used to generate the models in this paper was based on the one found here: https://github.com/uber-research/ape-x. The hyperparameters are reported in Table S3.

S3.3 GA

The implementation of GA used to generate the models in this paper was based on the one found here: https://github.com/uber-research/deep-neuroevolution. The hyperparameters are reported in Table S4 and were found through random search.

S3.4 ES

The implementation of ES used to generate the models in this paper was based on the one found here: https://github.com/uber-research/deep-neuroevolution. The hyperparameters reported in Table S5 were found via preliminary search and are similar to those reported in \citepNewconti:improving.

S3.5 Impala

The implementation of Impala used to generate the models in this paper is based on the one found here: https://github.com/deepmind/scalable_agent. The hyperparameters reported in Table S6 are the same as those reported in \citepNewespeholt:impala.

\bibliographystyleNew

named \bibliographyNewcites