Dynamic Planning Networks

TL;DR

Dynamic Planning Networks (DPN) integrate model-based and model-free reinforcement learning to enable efficient online planning, significantly reducing the number of state-transitions needed and demonstrating improved performance and generalization.

Contribution

DPN introduces a novel architecture that dynamically constructs plans using learned models, combining search strategies with deep learning for improved RL efficiency.

Findings

Reduces state-transitions during planning by up to 96%

Emergent classical search patterns like BFS and DFS

Improves data efficiency, performance, and generalization

Abstract

We introduce Dynamic Planning Networks (DPN), a novel architecture for deep reinforcement learning, that combines model-based and model-free aspects for online planning. Our architecture learns to dynamically construct plans using a learned state-transition model by selecting and traversing between simulated states and actions to maximize information before acting. In contrast to model-free methods, model-based planning lets the agent efficiently test action hypotheses without performing costly trial-and-error in the environment. DPN learns to efficiently form plans by expanding a single action-conditional state transition at a time instead of exhaustively evaluating each action, reducing the required number of state-transitions during planning by up to 96%. We observe various emergent planning patterns used to solve environments, including classical search methods such as breadth-first and depth-first search. DPN shows improved data efficiency, performance, and generalization to new and unseen domains in comparison to several baselines.

Equations

$$\begin{array}{c}{\mathcal{U}}_{\tau }(h_{\tau +1}^{\mathcal{O}},h_{\tau }^{\mathcal{O}},z_{\tau })=V(z_{\tau +1})+\mathbf{D}[h_{\tau +1}^{\mathcal{O}},h_{\tau }^{\mathcal{O}}]\end{array}$$

$$h_{\tau }^{\mathcal{O}}=W^{zh}{z}_{\tau }$$

$$\begin{array}{c}{w}_{\tau }\sim \text{G}({\mathbf{W}}^{h3}{h}_{\tau }^{\mathcal{I}})\\ z_{\tau }^{\ast }=w_{\tau }[z_{\tau }^p,z_{\tau }^c,z_{\tau }^r]\end{array}$$

$$a_{\tau }^{\ast }\sim \text{G}({\mathbf{W}}^{azh}[z_{\tau }^{\ast },h_{\tau }^{\mathcal{I}}])$$

$$\begin{array}{c}{z}^{\prime }=z_{\tau }+tanh({\mathbf{W}}^{zz}{z}_{\tau }^{\ast })\\ z^{\prime \prime }=z^{\prime }+tanh((a_{\tau }^{\ast }{\mathbf{W}}^{azz})z^{\prime })\\ z_{\tau +1}^{\ast }=z_{\tau }+z^{\prime \prime }\end{array}$$

$$a_t={\mathbf{W}}^{ah}tanh({\mathbf{W}}^{hh}{h}_T^{\mathcal{I}}+h_{\tau =0}^{\mathcal{O}})$$

$$\Delta \theta ={\nabla }_{\theta }{\mathcal{L}}_{\mathcal{O}}+{\nabla }_{\theta }{\mathcal{L}}_{\mathcal{I}}+\lambda {\nabla }_{\theta }{\mathcal{L}}_{\mathcal{Z}}-\beta {\nabla }_{\theta }H$$

Full text

Machine Learning, ICML

1 Introduction

The central focus of reinforcement learning (RL) is the selection of optimal actions to maximize the expected reward in an environment where the agent must rapidly adapt to new and varied scenarios. Various avenues of research have spent considerable efforts improving core axes of RL algorithms such as performance, stability, and sample efficiency. Significant progress on all fronts has been achieved by developing agents using deep neural networks with model-free RL (Mnih et al., 2015, 2016; Schulman et al., 2015, 2017; OpenAI, 2018); showing model-free methods efficiently scale to high-dimensional state space and complex domains with increased compute. Unfortunately, model-free policies are often unable to generalize to variances within an environment as the agent learns a policy which directly maps environment states to actions.

A favorable approach to improving generalization is to combine an agent with a learned environment model, enabling it to reason about its environment. This approach, referred to as model-based RL learns a model from past experience, where the model usually captures state-transitions, $p(s_{t+1}|s_{t},a_{t})$, and might also learn reward predictions $p(r_{t+1}|s_{t},a_{t})$. Usage of learned state-transition models is especially valuable for planning, where the model predicts the outcome of proposed actions, avoiding expensive trial-and-error in the actual environment – improving performance and generalization. This contrasts with model-free methods which are explicitly trial-and-error learners (Sutton & Barto, 2017). Historically, applications have primarily focused on domains where a state-transition model can be easily learned, such as low dimensional observation spaces (Peng & Williams, 1993; Deisenroth & Rasmussen, 2011; Levine & Abbeel, 2014), or where a perfect model was provided (Coulom, 2006; Silver et al., 2016a) – limiting usage. Furthermore, application to environments with complex dynamics and high dimensional observation spaces has proven difficult as state-transition models must learn from agent experience, suffer from compounding function approximation errors, and require significant amounts of samples and compute (Oh et al., 2015; Chiappa et al., 2017; Guzdial et al., 2017). Fortunately, recent work has overcome the aforementioned difficulties by learning to interpret imperfect model predictions (Weber et al., 2017) and learning in a lower dimensional state space (Farquhar et al., 2017a).

Planning in RL has used state-transition models to perform simulated trials with various styles of state traversal such as: recursively expanding all available actions per state for a fixed depth (Farquhar et al., 2017a), expanding all actions of the initial state and simulating forward for a fixed number of steps with a secondary policy (Weber et al., 2017), or performing many simulated rollouts with each stopping when a terminal state is encountered (Silver et al., 2016a). An issue arises within simulated trials when correcting errors in action selection, as actions can either be undone by wasting a simulation step, using the opposing action, or are irreversible, causing the remaining rollout steps to be sub-optimal in value. Ideally, the agent can step the rollout backwards in time thereby undoing the poor action and choosing a better one in its place. Additionally, during rollouts the agent is forced to either perform a fixed number of steps or continue until a terminal state has been reached; when ideally a rollout can terminate early if the agent decides the path forward is of low value.

In this paper, we propose an architecture that learns to dynamically control a state-transition model of the environment for planning. By doing so, our model has greater flexibility during planning allowing it to efficiently adjust previously simulated actions. We demonstrate improved performance against both model-free and planning baselines on varied environments.

The paper is organized as follows: Section 2 covers our architecture and training procedure, Section 3 covers related work, Section 4 details the experimental design used to evaluate our architecture, and in Section 5 we analyze the experimental results of our architecture.

2 Dynamic Planning Network

In this section, we describe DPN, a novel planning architecture for deep reinforcement learning (DRL). We first discuss the architecture overview followed by the training procedure. Steps taken in the environment use subscript $t$ and steps taken during planning use subscript $\tau$.

2.1 DPN Architecture

The architecture is comprised of an inner agent, an outer agent, a shared encoder, and a learned state-transition model. Figure 1(a) illustrates a high-level diagram of the DPN architecture. The outer agent (OA) is a feed-forward network and the inner agent (IA) is based on a recurrent neural network (RNN). The architecture interacts with the environment by observing raw environment states $s_{t}\in\mathcal{S}$ and outputting actions $a_{t}\in\mathcal{A}$ via OA. However, before OA outputs an action $a_{t}$ the IA performs $\tau=1,...,T$ steps of planning by interacting with an internal simulated environment; where this simulated environment is defined by the state-transition model and sub-section of OA’s network. Selection of an action $a_{t}$ by OA uses the final hidden state $h^{\mathcal{I}}_{T}\in\mathbb{R}^{1\times h^{i}}$ of IA and an embedding of the current state $z_{t}\in\mathbb{R}^{1\times z}$. The objective of IA is to maximize the total planning utility provided to the OA; where planning utility, given in Equation 1, measures the value of the new state $z_{\tau+1}$ and the change in the OA’s hidden state if it were to have undergone a state transition from $z_{\tau}$ to $z_{\tau+1}$.

[TABLE]

where $z_{\tau+1}$ is the state transitioned to after performing an action $a_{\tau}$ in state $z_{\tau}$, $h^{\mathcal{O}}_{\tau}\in\mathbb{R}^{1\times h^{o}}$ and $h^{\mathcal{O}}_{\tau+1}$ are the hidden states of OA after perceiving the current state $z_{\tau}$ and state transitioned to $z_{\tau+1}$ respectively, $\mathbf{D}$ is a distance measure, and $V(z_{\tau+1})$ is the value OA assigns the next state $z_{\tau+1}$. After empirical evaluation of different distance functions, such as L2, Cosine distance, and KL, we found the L1 distance function to be the most performant.

The planning utility formulation is similar to previous work on intrinsic motivation (Plappert et al., 2017; Chentanez et al., 2005). Where agent rewards are usually defined as $r^{e}_{t}+r^{i}_{t}$ with $r^{e}_{t}$ representing the external reward given by the environment and $r^{i}_{t}$ is an internally generated reward. In the case of Equation 1, external reward is provided by the OA’s value function and the internal reward is defined by the L1 distance between OA’s internal representation of $z_{\tau}$ and $z_{\tau+1}$.

Therefore, to maximize planning utility for OA during each planning step $\tau$, IA must select appropriate simulated-states $z^{*}_{\tau}$ and actions $a^{*}_{\tau}$. A simulated-state $z^{*}_{\tau}$ is selected from one of three embedded states tracked during planning: the previous $z^{p}_{\tau}$, current $z^{c}_{\tau}$, and root states $z^{r}_{\tau}$; with the triplet written as $z^{\{p,c,r\}}_{\tau}$ for convenience. Initially, $z^{\{p,c,r\}}_{\tau=0}$ is set to an embedding $z_{t}$ produced by the encoder of the initial raw state $s_{t}$ as $z_{\tau=0}=encoder(s_{t})$. The encoder is comprised of a series of convolutional layers specific to each environment. Before planning begins, OA’s hidden state is updated using $z_{\tau=0}$:

[TABLE]

where ${\bm{W}}^{zh}\in\mathbb{R}^{z\times h^{o}}$ is a learnable parameter of OA with biases omitted. Within this work, we consider the intermediate activation from the OA, a feed-forward network, as a hidden state. The simulated-action $a^{*}_{\tau}$ mirrors those available to OA in the environment, such that $a^{*}_{\tau}\in\mathcal{A}$.

Empirically, we found that using the same policy for planning and acting caused poor performance. We hypothesize that the optimal policy for planning is inherently different from the one required for optimal control in the environment; as during planning, a bias toward exploration might be optimal.

2.2 A Planning Step

At each planning step $\tau$, shown in Figure 1(b), the IA selects a simulated-state $z^{*}_{\tau}$ and action $a^{*}_{\tau}$ by considering the previous hidden state $h^{\mathcal{I}}_{\tau-1}$, the triplet of embedded states $z^{\{p,c,r\}}_{\tau}$, a scalar representing the current planning step $\tau/T$, and OA’s hidden state $h^{\mathcal{O}}_{\tau}$ given $z^{c}_{\tau}$. The information is concatenated together forming a context and is fed into IA, a recurrent network producing an updated hidden state $h^{\mathcal{I}}_{\tau}\in\mathbb{R}^{1xh^{i}}$. The updated hidden state is used to select the simulated-state $z^{*}_{\tau}$ by multiplying $z^{\{p,c,r\}}_{\tau}$ with a 1-hot encoded weight $w_{\tau}\in\{0,1\}^{1\times 3}$ sampled from the Gumbel-Softmax distribution, G:

[TABLE]

where ${\bm{W}}^{h3}\in\mathbb{R}^{h^{i}\times 3}$ is a learnable parameter belonging to IA and G is the Gumbel-Softmax distribution (GSD). Where the GSD is a continuous relaxation of the discrete categorical distribution giving access to differentiable discrete variables (Jang et al., 2016; Maddison et al., 2016). Empirically, we found that using a 1-hot encoding for the weight $w_{\tau}$ gives greater performance than a softmax activation. Therefore, we used GSD in place of softmax activations throughout our architecture. Next, the simulated-action $a^{*}_{\tau}\in\{0,1\}^{1\times\mathcal{A}}$, is sampled as follows:

[TABLE]

where ${\bm{W}}^{azh}\in\mathbb{R}^{a\times z+h^{i}}$ is a learnable parameter of IA. In Equation 4 the selected simulated-state $z^{*}_{\tau}$ and IA’s hidden state $h^{\mathcal{I}}_{\tau}$ are concatenated, passed through a linear layer, and used as logits for GSD. Then, with the selected simulated-state $z^{*}_{\tau}$ and simulated-action $a^{*}_{\tau}$, we produce the next state $z_{\tau+1}$ using the state-transition model, defined as:

[TABLE]

where ${\bm{W}}^{zz}\in\mathbb{R}^{z\times z}$ and ${\bm{W}}^{azz}\in\mathbb{R}^{\mathcal{A}\times z\times z}$ are learnable parameters of the state-transition model. We parameterize each available action in $\mathcal{A}$ with a learned weight matrix that carries information about the effect of taking an action $a^{*}_{\tau}\in\mathbb{R}^{1\times\mathcal{A}}$. We use the same state-transition model presented by Farquhar et al. (2017a). Finally, the three embedded states are updated as: $z^{p}_{\tau+1}=z^{c}_{\tau}$, $z^{r}_{\tau+1}=z_{\tau=0}$, and $z^{c}_{\tau+1}=z^{*}_{\tau+1}$.

2.3 Action selection

The IA repeats the process defined in Section 2.2 of selecting $z^{*}_{\tau}$ and $a^{*}_{\tau}$ for $T$ steps before finally emitting a final hidden state $h^{\mathcal{I}}_{T}$ summarizing the result of planning. The OA uses IA’s final hidden state $h^{\mathcal{I}}_{T}$ and its initial hidden state $h^{\mathcal{O}}_{\tau=0}$ to select an action $a_{t}$:

[TABLE]

where ${\bm{W}}^{hh}\in\mathbb{R}^{h^{o}\times h^{i}}$ and ${\bm{W}}^{ah}\in\mathbb{R}^{\mathcal{A}\times h^{o}}$ are learnable parameters of OA. Finally, the hidden state of the IA is reset.

2.4 Tree Interpretation

The planning process can be interpreted as dynamically expanding a state-action tree, illustrated in Figure 2, where all edges and vertexes are chosen by IA to maximize the total planning utility provided to OA.

With simulated-state selection $z^{*}_{\tau}$, using $w_{\tau}$, IA controls which node in the tree is expanded further: the parent node ($z^{p}$), the root node ($z^{r}$), or the current node ($z^{c}$). While action selection $a^{*}$ chooses the branching direction, exploring the embedded state space using the state-transition model.

The illustration of a constructed tree in a fictional environment is shown in Figure 2. State selections are shown in light purple, and state transitions with an action, using the state-transition model, are shown as blue. The source state is shown as a grey circle with a blue outline and the transitioned state is shown as a fully blue circle. In this example, there are three actions, each corresponding to their graphical representation: left, right, and down. The root state is marked with an ?R?. An example of a possible tree construction for T=3 steps of planning: a) step $\tau$ = 1, IA selects the current state $z^{c}$ and transitions to a new state with action ?left?; b) step $\tau$ = 2, the IA selects the current state $z^{c}$ and ?down? action; and step $\tau=3$ c) IA selects the root state $z^{c}$ and ?down? action.

2.5 Training Procedure

We trained the parameters of DPN using A2C, a single threaded version of asynchronous advantage actor-critic (A3C) (Mnih et al., 2016). The OA was trained as an actor-critic, with policy and value networks, while the IA was trained as a policy network only. As defined in Equation 1, IA used OA’s value network as part of its target. A Huber loss was added to the loss function to perform state-grounding of the state-transition model between the current state $z_{t}$, action $a_{t}$, and $z_{t+1}$ which we denoted with $\mathcal{L}_{\mathcal{Z}}$. Combining our losses, the architecture is trained using the following gradient:

[TABLE]

where $\mathcal{L}_{\mathcal{O}}$ is the OA loss (policy and value), $\mathcal{L}_{\mathcal{I}}$ is the IA policy loss, $\lambda$ controls the state-grounding loss, $H$ is the entropy regularizer computed for OA and IA, and $\beta$ is hyperparameter tuning entropy maximization of all policies; we used the same $\beta$ value for each policy. The losses $\mathcal{L}_{\mathcal{O}}$ and $\mathcal{L}_{\mathcal{Z}}$ are computed over all parameters; while $\mathcal{L}_{\mathcal{I}}$ and its entropy regularizer losses are computed with respect to only IA’s parameters. We perform updates to IA in this way as to stop IA from cheating by modifying the parameters of the OA that define its reward via $\mathbf{D}[h^{\mathcal{O}}_{\tau+1},h^{\mathcal{O}}_{\tau}]$ and $V(z_{\tau+1})$ within the planning utility.

We used 16 workers with RMSprop optimizer (Tieleman & Hinton, 2012) across all environments.

3 Related Work

Various efforts have been made to combine model-free and model-based methods, such as the Dyna-Q algorithm (Sutton, 1991) that learns a model of the environment and uses this model to train a model-free policy. Originally applied in the discrete setting, Gu et al. (2016) extended Dyna-Q to continuous control. In a similar spirit to the Dyna algorithm, recent work by Ha & Schmidhuber (2018) combined data generated from a pre-trained unsupervised model with evolutionary strategies to train a policy. However, none of the aforementioned algorithms use the learned model to improve the online performance of the policy and instead use the model for offline training. Therefore, the learned models are typically trained with a tangential objective to that of the policy such as a high-dimensional reconstruction. In contrast, our work learns a model in an end-to-end manner, such that the model is optimized for its actual use in planning.

Guez et al. (2018) proposed MCTSnets, an approach for learning to search where they replicate the process used by MCTS. MCTSnets replaces the traditional MCTS components by neural network analogs. The modified procedure evaluates, expands, and back-ups a vector embedding instead of a scalar value. The entire architecture is end-to-end differentiable.

Tamar et al. (2016) trained a model-free agent with an explicit differentiable planning structure, implemented with convolutions, to perform approximate on-the-fly value iteration. As their planning structure relies on convolutions, the range of applicable environments is restricted to those where state-transitions can be expressed spatially.

Pascanu et al. (2017) implemented a model-based architecture comprised of several individually trained components that learn to construct and execute plans. Their work used a single policy for planning and acting while DPN has a separate policy for each mode. The policy used in DPN for planning also selects which state to plan from while Pascanu et al. (2017) used a separate specialized policy. They examine performance on Gridworld tasks with single and multi-goal variants but on an extremely limited set of small maps.

Vezhnevets et al. (2016) proposed a method which learns to initialize and update a plan; their work does not use a state-transition model and maps new observations to plan updates.

Value prediction networks (VPNs) by Oh et al. (2017), Predictron by Silver et al. (2016b), and Farquhar et al. (2017a), an expansion of VPNs, combine learning and planning by training deep networks to plan through iterative rollouts. The Predictron predicts values by learning an abstract state-transition function. VPNs constructs a tree of targets used only for action selection. Farquhar et al. (2017a) create an end-to-end differentiable model that constructs trees to improve value estimates during training and acting. Both Oh et al. (2017) and (Farquhar et al., 2017a) construct plans using forward-only rollouts by exhaustively expanding each state’s actions. Similarly, François-Lavet et al. (2018) proposed a model that combined model-free and model-based components to plan on embedded state representations in a similar fashion to TreeQN (Farquhar et al., 2017a). They propose an additional loss to the objective function, an approximate entropy maximization penalty, that ensures the expressiveness of the learned embedding.

In contrast to the aforementioned works, during planning DPN learns to selectively expand actions at each state, with the ability to adjust sub-optimal actions, and uses planning results to improve the policy during both training and acting.

Weber et al. (2017) proposed Imagination Augmented Agents (I2As), an architecture that learns to plan using a separately trained state-transition model. Planning is accomplished by expanding all available actions $\mathcal{A}$ of the initial state and then performing $\mathcal{A}$ rollouts using a tied-policy for a fixed number of steps. In contrast, our work learns the state-transition model end-to-end, uses a separate policy for planning and acting, and is able to dynamically adjust planning rollouts. Additionally, in terms of sample efficiency, I2As require hundreds of millions of steps to converge, with the Sokoban environment taking roughly 800 million steps. Though not directly comparable, our work in the Push environment, a puzzle game very similar to Sokoban, requires an order of magnitude fewer steps, roughly 20 million, before convergence.

Within continuous control learning a state-transition model for planning has been used in various ways. Finn & Levine (2017) demonstrate the usage of a predictive model of raw sensory observations with model-predictive control (MPC) where the model is learned in an entirely self-supervised manner. Srinivas et al. (2018) proposed using an embedded differential network that performs iterative planning through gradient descent over actions to reach a specified target goal state within a goal-directed policy. Henaff et al. (2017) focus on model-based planning in low-dimensional state spaces and extend their method to perform in both discrete and continuous action spaces.

Additional connections between learning environment models, planning and controls, and other methods related to ours were previously discussed by Schmidhuber (2015).

4 Experiments

We evaluated DPN on a multi-goal Gridworld environment and Push, (Farquhar et al., 2017a) a box-pushing puzzle environment. Push is similar to Sokoban used by Weber et al. (2017) with comparable difficulty. Within our experiments, we evaluated our model performance against either model-free baselines, DQN and A2C, or planning baselines, such as TreeQN and ATreeC. The experiments are designed such that a new scenario is generated across each episode, which ensures that the solution of a single variation cannot be memorized. We are interested in understanding how well our model can adapt to varied scenarios. Additionally, we investigate how planning length $T$ affects model performance, how IA branching during planning affects performance, different distance functions for the IA’s reward function, and planning patterns that our agent learned in the Push environment. Full details of the environments, experimental setup, hyperparameters are provided in the supplemental material. Unless specified otherwise, each model configuration is averaged over 3 different seeds and is trained for 40 million steps.

Push: The Push environment is a box-pushing domain, where an agent must push boxes into goals while avoiding obstacles, with samples shown in Figure 4(a). Since the agent can only push boxes, with no pull actions, poor actions within the environment can lead to irreversible configurations. The agent is randomly placed, along with 12 boxes, 5 goals, and 6 obstacles on the center 6x6 tiles of an 8x8 grid. Boxes cannot be pushed into each other and obstacles are ?soft? such that they do not block movement, but generate a negative reward if the agent or a box moves onto an obstacle. Boxes are removed once pushed onto a goal. We use the open-source implementation provided by Farquhar et al. (2017b). The episode ends when the agent collects all goals, steps off the map, or goes over 75 steps. We compare our model performance against planning baselines, TreeQN and ATreeC (Farquhar et al., 2017a), as well as model-free baselines, DQN (Mnih et al., 2015) and A2C (Mnih et al., 2016).

Gridworld: We use a Gridworld domain with randomly placed obstacles that an agent must navigate searching for goals. The environment, randomly generated between episodes, is a 16x16 grid with 3 goals. We force a minimum distance between goals and between the agent and goals. The agent must learn an optimal policy to solve new unseen maps. Figure 6(a) shows several instances of a 16x16 Multi-goal Gridworld. The rewards that an agent receives are as follows: +1 for each goal captured, -1 for colliding with a wall, -1 for stepping off the map, -0.01 for each step, and -1 for going over the step limit. An episode terminates if the agent collides with an obstacle, collects all the goals, steps off the map, or goes over 70 steps. We evaluate our algorithm against model-free baselines such as A2C (Mnih et al., 2016) and variants of DQN (recurrent and non-recurrent) (Mnih et al., 2015). Each baseline used the same encoder structure as DPN. We train for 20 million environment steps.

Planning length: Using the Push environment, we varied the parameter $T$, which adjusts the number of planning steps, with $T=\{1,2,3\}$ evaluated. The Push environment was chosen because the performance is sensitive to an agent’s ability to plan effectively.

Inner Agent Branching: We examine the affect on performance of different branching options for $T=3$: current, reset, or all. We also included the ATreeC-1 baseline as this corresponds to the reset branching option of our architecture and serves as a sanity check.

Inner Agent Distance Functions: We vary the distance function used by the IA’s loss defined in Equation 1. We examine L1, L2, KL, and Cosine distance functions.

Planning Patterns: We examine the planning patterns that our agent learns in the Push environment for T=3 steps. Here we are interested in understanding what information the agent extracts from the simulation as context before acting.

5 Results and Discussion

5.1 Push Environment

Figure 4(c) shows DPN, with planning length $T=3$, compared to DQN, A2C, TreeQN and ATreeC baselines 111The data for the training curves of DQN, A2C, TreeQN, and ATreeC were provided by Farquhar et al. via email correspondence. Each experiment was run with 12 different seeds for 40million steps.. For TreeQN and ATreeC, we chose tree depths which gave the best performance, corresponding to tree depths of 3 and 1 respectively. Our model clearly outperforms both planning and non-planning baselines: TreeQN, ATreeC, DQN, and A2C. We see that our architecture converges at a faster rate than the other baselines, matching ATreeC-1’s final performance after roughly 20 million steps in the environment. In comparison to the other planning baselines, TreeQN and ATreeC, require roughly 35-40 million steps: $\sim$2x additional samples.

We note that the planning efficiency of DPN is higher in terms of overall performance per number of state-transitions. On the Push environment, with $\mathcal{A}=4$ actions, TreeQN with tree depth of $d=3$ requires \big{(}\frac{\mathcal{A}^{d+1}-1}{\mathcal{A}-1}\big{)}-1=84 state-transitions. In contrast, DPN with planning length of $T=3$ requires only $T$ state-transitions – a 96% reduction. Loosely comparing to I2As, simply in terms of state-transitions, we see that I2As require $\mathcal{A}\times L$ state-transitions per action step, where $L$ is the rollout length. This performance improvement is a result of DPN learning to selectively expand actions and being able to dynamically adjust previously simulated actions during planning.

5.2 Multi-Goal Gridworld

Figure 6(c) shows, the results of DPN compared to various model-free baselines. Within this domain, the difference in performance is clear: our model outperforms the baselines by a significant margin. The policies that DPN learns generalizes better to new scenarios, can effectively avoid obstacles, and is able to capture multiple goals. Of the model-free baselines, we see that the A2C baseline performs the best. We believe that the A2C baseline is able to better explore the environment due to the multiple workers running in parallel throughout training. Additionally, as seen in Figure 6(c), the DQN variants fail to capture any goals and do not achieve a score higher than -1.0. This indicates the DQN baselines learn only to navigate around the map without collecting goals before an episode ends. It should be noted we saw little performance improvement even when allocating the model-free algorithms an additional 2x environment steps (40 million) or, in the case of DQN models, a 2-4x longer exploration period (8-16 million).

The poor performance of the baseline models might be the result of high variance in the environment’s configuration between episodes. We believe that DPN performs better because it captures common structure present between all permutations of the environment by using the environment model. This allows it to exploit the model for planning in newly generated mazes.

5.3 Planning length

In Figure 9, we see the performance of our model over the planning lengths $T=\{1,2,3\}$. As seen in Figure 9, model performance increases as we add additional planning steps, while the number of model parameters remains constant.

As the planning length increases, we see the general trend of faster model convergence. Even a single step $T=1$ of planning allows the agent to test action-hypotheses and avoid poor choices in the environment. From Figure 9 we see that an additional planning step, from $T=1$ to $T=2$, does not provide benefit until later in training. The planning length $T=3$ provides the best performance and faster convergence. We suspect that the shorter planning lengths do not allow the IA to learn a policy that provides enough utility to OA. Ideally, the architecture would be able to adjust the number of planning steps $T$ dynamically based on current needs. We see this expansion, similar to the adaptive computation presented by Graves (2016), as an interesting avenue for future work.

5.4 Inner Agent Distance Functions

From 7(a), we see an evaluation of distance functions used in Equation 1. The L1 distance function has the best performance with slightly faster convergence. While the L2, Cosine, and KL functions have worse performance. We hypothesize that the L1 distance performed better due to its robustness to outliers, a likely event during learning, as the distance is a function of noisy and changing vectors from the OA and state-transition model.

5.5 Inner Agent Branching

In Figure 7(b), we see how different branching options affects the architecture performance. Our proposed branching improves performance of the architecture as compared to the current and reset options. Interestingly, the performance of our architecture when using the reset options is roughly the same as ATreeC-1. This is unsurprising as the reset and ATreeC-1 options employ a similar planning strategy of a shallow 1-step look-ahead. The small discrepancy in performance could be due to ATreeC-1 evaluating all 4 actions while DPN evaluates only 3. We see that the current branching option results in better performance and is amounts to a forward-only rollout of length $T=3$. We hypothesize the performance difference between current and reset is from DPN being able to see the results of its actions from the first planning step over a longer time span.

5.6 Planning Patterns

By watching a trained agent play through newly generated maps, we identified common planning patterns, which are shown in Figure 8. Two prominent patterns emerged: breadth-first search and depth-first search. From Figure 8(a) we can see that our agent learned to employ breadth-first search, where planning steps are used to expand available actions around the agent, corresponding to a tree of depth 1. In contrast, depth-first search as seen in Figure 8(b), has the agent expanding state forward only. The agent does not always follow depth-first search paths and seems to use them to ?check? if a particular pathway is worth pursuing.

6 Conclusion

In this paper, we have presented DPN, a new architecture for deep reinforcement learning that uses two agents, IA and OA, working in tandem. Empirically, we have demonstrated that DPN outperforms the model-free and planning baselines in both the mutli-goal Gridworld and Push environments while using $\sim$2x fewer environment samples. Ablation studies have shown that our proposed target for the IA improves performance and helps increase the speed of convergence. We have shown that the IA learns to dynamically construct plans that maximize utility for the OA; with IA learning to dynamically use classical search patterns, such as depth-first search, without explicit instruction. Compared to other planning architectures, DPN requires significantly fewer state-transitions during planning for the same level of performance – drastically reducing computational requirements by up to 96%.

Acknowledgments

We would like to thank Eder Santana, Tony Zhang, and Justin Tomasi for helpful feedback and discussion. This project received funding from Ontario Centres of Excellence, Voucher for Innovation and Productivity (“VIPI”) Program Project #29393.