Harnessing Reinforcement Learning for Neural Motion Planning

TL;DR

This paper introduces DDPG-MP, a reinforcement learning approach tailored for neural motion planning that outperforms supervised methods and accelerates planning in new domains by leveraging learned policies and known models.

Contribution

We propose DDPG-MP, a modified reinforcement learning algorithm that exploits known models and solved plans to improve neural motion planning accuracy and speed.

Findings

RL methods mitigate boundary data issues better than supervised learning.

DDPG-MP significantly improves motion planning accuracy.

Our method enables faster planning in new domains with sufficient training data.

Abstract

Motion planning is an essential component in most of today's robotic applications. In this work, we consider the learning setting, where a set of solved motion planning problems is used to improve the efficiency of motion planning on different, yet similar problems. This setting is important in applications with rapidly changing environments such as in e-commerce, among others. We investigate a general deep learning based approach, where a neural network is trained to map an image of the domain, the current robot state, and a goal robot state to the next robot state in the plan. We focus on the learning algorithm, and compare supervised learning methods with reinforcement learning (RL) algorithms. We first establish that supervised learning approaches are inferior in their accuracy due to insufficient data on the boundary of the obstacles, an issue that RL methods mitigate by actively exploring the domain. We then propose a modification of the popular DDPG RL algorithm that is tailored to motion planning domains, by exploiting the known model in the problem and the set of solved plans in the data. We show that our algorithm, dubbed DDPG-MP, significantly improves the accuracy of the learned motion planning policy. Finally, we show that given enough training data, our method can plan significantly faster on novel domains than off-the-shelf sampling based motion planners. Results of our experiments are shown in https://youtu.be/wHQ4Y4mBRb8.

Tables

Table 1. TABLE I : Success rates simple scenario

Model	Test	Validation
DDPG	865	0.82533
DDPG-MP (no expert)	1.0	0.99366

Table 2. TABLE II : Success rates hard scenario

Model	Test	Validation
DDPG	0.36833	0.31766
DDPG+HER	0.3633	0.286
DDPG-MP (no expert)	0.83	0.81933
DDPG-MP+HER (no expert)	0.72833	0.68866
DDPG-MP (full)	0.99	0.9733

Table 3. TABLE III : Success rates vision scenario

Model	Test	Validation
DDPG	0.9233	0.89733
DDPG+HER	0.955	0.925
DDPG-MP (full)	0.96166	0.93566

Table 4. TABLE IV : Validation success rates Behavioral-Cloning by number of motion plans demonstrations seen

Number of episodes seen	Validation success rate
10K	0.725
20K	0.67
40K	0.73
80K	0.76
120K	0.789
160K	0.775

Equations

$$T(0)=c_0,FK(T(1))=g,\forall a\in [0,1]:\varphi (T(a))=True.$$

$$\begin{array}{rl}{Q}^{\pi }& (s_t,a_t)=\\ & {\mathbb{E}}_{s_{t+1}\sim p}\left[r(s_t,a_t)+\gamma {\mathbb{E}}_{a_{t+1}\sim \pi (s_{t+1})}\left[Q^{\pi }(s_{t+1},a_{t+1})\right]\right].\end{array}$$

$${\nabla }_{\theta }J\approx {\mathbb{E}}_{s\sim \beta }\left[{\nabla }_{\theta }\hat{Q}(s,a)\right],$$

$$c_{t+1}=c_t+a_t.$$

$$s_{t+1}=\{w,c_t+a_t\}\doteq f(s_t,a_t).$$

$$r_t^T=\{\begin{array}{ll}-ϵ,& (c_t,c_{t+1})\in T_{free},\\ 1,& (c_t,c_{t+1})\in T_{goal},\\ -1,& (c_t,c_{t+1})\in T_{col}.\end{array}$$

$$\begin{array}{rl}{Q}^{\pi }(s_t,a_t)=& r(s_t,a_t)+\cdots +{\gamma }^{k-1}r(s_{t+k},\pi (s_{t+k}))\\ & +{\gamma }^k{Q}^{\pi }(s_{t+k+1},\pi (s_{t+k+1})).\end{array}$$

$$\begin{split}\nabla_{\theta}J=&\mathbb{E}_{s\sim\beta}\big{[}\nabla_{\theta}\big{(}r(s_{t},\pi(s_{t}))\\ &+\gamma\mathbb{I}_{(c_{t},c_{t+1})\in T_{free}}Q(f(s_{t},\pi(s_{t})),\pi(f(s_{t},\pi(s_{t}))))\big{)}\big{]}\end{split}$$

$$\frac{1}{N}\sum _i{▽}_{\theta }\left(\tilde{r}(w,c_i,a_i)+\gamma \tilde{p}(w,c_i,a_i)\hat{Q}(w,c_i,a_i)\right)$$

$${\theta }^{{\hat{Q}}^{\prime }}\leftarrow \tau {\theta }^{\hat{Q}}+(1-\tau ){\theta }^{{\hat{Q}}^{\prime }}$$

$${\theta }^{{\pi }^{\prime }}\leftarrow \tau {\theta }^{\pi }+(1-\tau ){\theta }^{{\pi }^{\prime }}$$

Full text

I Introduction

Motion planning – the problem of finding a collision free trajectory for a robot – is a fundamental component in almost all robotic applications deployed today [28, 27]. Sampling based motion planners such as probabilistic roadmaps [22] and rapidly exploring random trees [27] have been studied extensively, can be guaranteed to converge to an optimal solution [21], and are common practice in various robotic domains and off-the-shelf software implementations [46, 45]. However, for domains where the environment can change rapidly, such as in e-commerce applications, industry 4.0 [26], or home robotics, it is desired to plan fast, and the computational burden of sampling based planners can be limiting.

Consider, for example, planning pick-and-place motions for a robot in an e-commerce warehouse that handles various products. Building on the insight that the changes in the task will mostly be limited, e.g., just the product will change, while the robot and surrounding will stay the same, several recent studies have explored a learning setting for motion planning, where data from previously solved motion plans is used to speed-up planning in new domains [6, 20, 10, 12].

In recent years, deep learning has proven capable of learning complex patterns in data for various decision making domains such as computer vision, protein folding, and games [24, 14, 43]. Motivated by these successes, we focus here on approaches that we collectively term neural motion planners [35, 5, 50, 38, 39], which use deep learning to approximate a motion planning computation. In a neural motion planner, a deep neural network is trained to map features of the domain (e.g., an image), the current robot state, and a goal robot state to the next robot state in the motion plan. By training on a set of motion planning domains, the network is hypothesized to learn the patterns which make for a successful motion plan, and, once trained, such a network can be used to quickly predict a motion plan in novel domains without running a heavy motion planning computation.

In this work, we investigate the algorithmic aspects of training a neural motion planner to solve nontrivial motion planning problems. We first consider a supervised learning approach, where previous plans are simply imitated [39, 16]. We observe that for high-dimensional domains that require high precision, the success of this approach is limited, which we attribute to insufficient data distribution on the boundary of the obstacles. We propose that reinforcement learning (RL; [47]) has the potential to overcome this problem, since the exploration process in RL will naturally drive the agent to investigate important areas in the domain, such as obstacle boundaries.

While RL algorithms are known to require extensive computation, the motion planning problem presents several features that can be exploited to dramatically improve RL performance. Our main contribution is an RL algorithm that exploits two features of motion planning problems: the fact that a model of the dynamics is known, and the fact that we can collect demonstration data offline through sampling based planners, to reduce the variance in training, and to perform efficient exploration.

We show that our method leads to significantly better accuracy, and much faster training times. In particular, we demonstrate predicting motion plans with almost perfect accuracy on a 4-dimensional robotic arm domain with very challenging narrow passages. With enough training time, our method learns to plan significantly faster than off-the-shelf sampling based planners. Our results suggest that with suitable training algorithms, deep learning can provide competitive results for learning in motion planning, opening the door to further investigations of network architectures and combinations of planning and learning.

II Background

We start by describing several mathematical frameworks which will be useful when we present our algorithm in Section V, such as the motion planning problems, and several learning frameworks that could be used to solve it.

II-A Motion Planning

We follow a standard motion planning formulation [28]. Let $C$ denote the configuration space (joint space) of the robot, let $S$ denote the task space (position and orientation) of the end effector, and let $FK:C\rightarrow S$ be the robot’s forward kinematics. Let $F\subset C$ denote the free space, and $\phi:C\rightarrow\{True,False\}$ a collision predicate that returns $True$ if $c\in F$. In the motion planning problem, the robot starts from a joint position $c_{0}\in C$, and is required to produce a trajectory plan $T:[0,1]\rightarrow C$ in the free space that ends with the end-effector at a goal pose $g\in S$:

[TABLE]

Popular motion planning algorithms rely on sampling the workspace, and effectively return a list of configurations $c_{0},\dots,c_{T}$ which make up the motion plan. A continuous plan is then derived by interpolation or smoothing [27]. We denote by $W=\{F,c_{0},g\}$ a tuple that describes the workspace – the parameters of a motion planning problem for a given robotic domain.

II-B Reinforcement Learning

In Reinforcement Learning (RL), an agent interacts with an environment $E$ by observing the current state and executing an action in discrete time steps. Once an action is executed the state changes (the change can be either deterministic or stochastic), the agent receives a reward and the process repeats until termination. This can be modeled as a Markov Decision Process (MDP [47]) with a state space $S$, action space $A$, an initial state distribution $\rho(s_{1})$, transition dynamics $p(s_{t+1}|s_{t},a_{t})$, and scalar reward function $r(s_{t},a_{t})$.

The goal of the agent is to interact with $E$ by selecting actions according to some policy $\pi(a|s)$ (possibly stochastic) which maximizes the $\gamma$-discounted future rewards defined for time step $t$ as $R_{t}=\mathbb{E}\left[\sum_{i=t}^{\infty}{\gamma^{i-t}r(s_{i},a_{i})}\right]$. Let $Q^{\pi}(s,a)$ be the expected discounted future reward obtained by following policy $\pi$, after executing action $a$ from state $s$: $Q^{\pi}(s,a)=\mathbb{E}\left[R_{t}|s_{t}=a,a_{t}=a,\pi\right]$. An important relation between current and future $Q^{\pi}$ is the Bellman equation:

[TABLE]

Let $Q^{*}(s,a)$ be the maximal expected discounted reward obtained by any strategy, after executing action $a$ from state $s$: $Q^{*}(s,a)=\max_{\pi}{\mathbb{E}[R_{t}|s_{t}=a,a_{t}=a,\pi]}$. For the optimal policy $\pi^{*}$ we have $Q^{\pi^{*}}(s,a)=Q^{*}(s,a)$ for all states and actions. The goal of an RL algorithm is to find such a policy.

II-C Actor-Critic Algorithm: DDPG

Actor-critic is a popular RL approach [47]. The actor $\pi(s|a)$ models a policy, while the critic $\hat{Q}(s,a)$ approximates the state-action value of the actor. The learning process interleaves improving the critic (for example, by using temporal difference learning [47]), with updating the policy towards actions with higher state-action values.

Deep Deterministic Policy Gradient (DDPG [31]) is an actor-critic RL algorithm designed for continuous state and action spaces, where $\pi$ and $\hat{Q}$ are modeled using neural networks, and $\pi$ is a deterministic policy with weights $\theta$. DDPG is an off-policy algorithm: it learns by observing actions taken by a stochastic exploration policy $\beta$, and storing the observed transitions in a replay buffer [33]. DDPG alternates between learning $\hat{Q}$ and learning $\pi$, fixing one while training the other. Specifically, the update of $\pi$ ascends in the direction which maximizes $\hat{Q}$, motivated by the policy gradient theorem:

[TABLE]

where $J$ denotes the expected discounted return [31].

II-D Hindsight Experience Replay

In NMP, the learned policy is goal-conditioned, i.e., takes in a goal as a parameter. The recent Hindsight Experience Replay (HER) [4], is an extension of off-policy RL methods such as DDPG to goal-conditioned policies. The idea is that, upon failing an episode (not reaching the goal), the agent adds to the replay-buffer an episode comprised of the same states and actions seen in the failed episode, but with the original goal replaced with the last encountered step – making the episode a success in hindsight. By making success more frequent, the agent is trained by an implicit curriculum, that eventually learns to cover the goal space. Indeed, HER has led to state-of-the-art results in goal-conditioned RL [4, 44, 30].

II-E Imitation Learning

In imitation learning [41], a decision making policy is learned from observing expert behavior. Following the RL notation, given a data set of $N$ expert demonstrations $\{s^{exp}_{i},a^{exp}_{i}\}_{i=1}^{N}$, the problem is to find a policy $\pi=P(a|s)$ that imitates the expert behavior. A popular imitation learning method is behavioral cloning [37], where $\pi$ in learned by maximizing the data log likelihood $\sum_{i=1}^{N}\log\pi(a^{exp}_{i}|s^{exp}_{i})$.

DAgger [40] is an imitation learning algorithm that corrects the distribution mismatch problem, where the distribution of states in the learning corpus may be different than the distribution of states encountered by the learned policy. In DAgger, after training a policy using behavioral cloning, the policy is executed, and additional expert data is first collected for all states visited, and then added to the agent’s data-set. Afterwards, a new policy is trained on the updated data-set and the process repeats to convergence.

III Problem Formulation

We consider the problem of learning for motion planning. In this setting, we assume that motion planning workspaces are generated from some (unknown) distribution $P(W)$. We are given a training data set of $N$ workspaces $\{W_{i}\}_{i=1}^{N}$ drawn from $P$, and the goal is to use this data to plan faster in an unseen test workspace $W_{test}$, also drawn from $P$.

We focus on an approach termed neural motion planning (NMP) [39]. In NMP, a workspace $W$ is described by a set of features, or context vector, $w$. The goal in NMP is to train a neural network policy $\pi$ that maps $w$ and the current configuration of the robot $c_{t}$ to the next step in a motion plan $c_{t+1}$. Once trained, such a network can produce a motion plan by unrolling the predictions to produce a list of states $c_{0},\dots,c_{T}$. We say that the motion plan is successful if it satisfies the motion planning problem conditions, i.e., $FK(c_{T})=g$, and $\phi(c_{t})=True\quad\forall 0\leq t\leq T$.111In practice, we can only verify that $\|FK(c_{T})-g\|<\epsilon$, for a small $\epsilon$. In the remainder of this work, we investigate how to best train the neural network in NMP.

IV Imitation Learning vs. RL

A straightforward method for training an NMP is using imitation learning. In this approach, a standard motion planner (e.g., RRT∗) is used to solve all the training work spaces. These motion plans are used to train a neural network using, e.g., behavioral cloning. This approach has been shown to work well for discrete planning problems [16, 9], and recently has been investigated in the context of NMP [39, 5, 50].

In our investigation, we found that for domains that require high precision in the motion plan, such as a robot moving through tight passages, the performance of imitation learning is severely limited (see explicit results in Section VII). We attribute this finding to the distribution of the training data, as depicted in figure 2 for the end effector of a robot arm moving in a narrow passage. In NMP, it is fundamental that the neural network learns not to hit obstacles. However, since the expert produces perfect motion plans, there is no information about hitting an obstacle in the training data. Note in Figure 2 that the data distribution for imitation learning does not cover the obstacle boundaries, thus there is no reason to expect that a network trained with this data will learn not to hit obstacles. This is an instance of the distribution mismatch problem described by Ross et al. [40]. While in principle the DAgger algorithm [40] could mitigate this issue, it is too costly to compute in practice as it requires running motion planning on every sample in the data. One can instead acquire expert data only for states that resulted in a collision. However, as we report in the supplementary material (Section -C), this approach did not lead to a significant improvement, which we believe is due to theses samples not being well-balanced with the rest of the data.

We propose that RL is more suitable for training NMPs. In RL, the agent starts out with a policy that does not solve the task, and through trial and error it gradually collects data on how not to hit obstacles, thereby improving its policy. Indeed, as depicted in Figure 2, the data distribution in RL has much more presence near the obstacle boundaries – the critical decision points for motion planning. Therefore, in this work we pursue an RL approach to NMP, as we describe next.

V RL for Motion Planning

In this section we first describe an RL formulation for NMP, and then propose a specialized RL algorithm for this domain.

We consider an episodic RL setting [47], where at each episode a workspace $W$ is randomly selected from the training set, and the task is to plan a successful motion plan in it. At time step $t$ of an episode, the robot configuration is $c_{t}$ and the context vector for $W$ is $w$. We define the state $s_{t}$ to be the tuple $\{w,c_{t}\}$, and the action to be the difference in the robot configuration:222Our work can be extended to kinodynamic motion planning by including the robot’s dynamics. We defer this to future work.

[TABLE]

Note that this implies deterministic dynamics, since the context vector $w$ is assumed to remain constant. The next state is:

[TABLE]

Upon hitting an obstacle, or after reaching the goal, the episode terminates. We define a reward such that maximizing it produces a correct solution to the motion planning problem, by encouraging movement toward the goal and not hitting obstacles. Note that any transition $\{c_{t},c_{t+1}\}$ can belong to one of three classes: free space movement, denoted as $T_{free}$; collision, denoted as $T_{col}$; and reaching the goal, denoted as $T_{goal}$. The transition reward is:

[TABLE]

The constant $\epsilon>0$ needs to be small enough such that the robot prefers moving in free space than colliding; we empirically set it as $0.01$.

Some motion planning problems may include additional rewards. For example, in our experiments we wanted to encourage the robot not waste effort by trying to move a joint beyond its limits. To model this, we use a reward of $r^{D}_{t}$ (domain specific reward), which in our case is the norm of the wasted movement (full details in section -B). Finally, the full reward of the system is $r_{t}=r^{T}_{t}+r^{D}_{t}$ .

V-A The DDPG-MP Algorithm

We next present Deep Deterministic Policy Gradient for Motion Planning (DDPG-MP) – a deep RL algorithm tailored for training an NMP. To motivate the algorithm, we observe that in the context of RL, the motion planning setting admits the following unique features:

The dynamics (Eq. (4)) and reward (Eq. (6)) are known. 2. 2.

Similar to imitation learning, we can obtain expert demonstrations for the work spaces in our training data.

Since our problem is deterministic, we build on the DDPG algorithm [31], a strong and popular deep RL algorithm for continuous control. We add to DDPG two modifications that exploit the motion planning features described above: we propose a model-based actor update that reduces variance, and use the expert demonstrations to perform efficient exploration. We next describe each idea in detail (for the DDPG-MP pseudo-code see Section -A in the supplementary material).

V-B Model-Based Actor Update

When training the actor network, the model is updated in the direction which maximizes $Q^{\pi}$. However, $Q^{\pi}$ is not known, and we only have access to an approximation $\hat{Q}^{\pi}$ learned by the critic, resulting in a high estimation errors in the policy gradient (due to variance and bias) of the actor. A key observation in our work is that we can reduce the variance in the actor update by using the known dynamics model. Note that in a deterministic domain with a deterministic policy, by definition (cf. Eq. (2)) we have that for any $k\in 0,1,\dots$:

[TABLE]

Thus, if we know the reward function and transition function, we can estimate $Q^{\pi}$ as $r(s_{t},a_{t})+\dots+\gamma^{k-1}r(s_{t+k},\pi(s_{t+k}))+\gamma^{k}\hat{Q}^{\pi}(s_{t+k+1},\pi(s_{t+k+1}))$, and we expect that as $k$ grows, the error in this approximation will reduce [17] (for $k\to\infty$ the error is zero). In our experiments, we found that $k=1$ is enough to significantly improve the actor update, and in the following we focus on this case. Extending our result to $k>1$ is straightforward.

Following the above derivation, keeping in mind that transitioning to a goal or obstacle terminates the episode, and using the specific form of the dynamics in our problem (5), we obtain the following actor update:

[TABLE]

Note, however, that both the indicator function in (7) and our reward function (6) are non differentiable, and their step-function structure means that the reward gradients in (7) will not be informative.

We therefore propose to use a smoothed reward function $\widetilde{r}$ and a smooth approximation $\widetilde{p}$ to $\mathbb{I}_{(c_{t},c_{t+1})\in T_{free}}$ instead. While one can derive explicit smoothing functions for these, in our work we opted for a general approach using supervised learning. Specifically, we represent $\widetilde{r}(w,c_{t},c_{t+1})$ and $\widetilde{p}(w,c_{t},c_{t+1})$ as neural networks, and train them using random samples of robot configurations. Technical implementation details regarding these models are given in Section -D of the supplementary material.333There are many possible choices for reward smoothing. We chose to model it as a neural network since it was easy to implement in out setup.

V-C Targeted Exploration via Expert Demonstrations Scoring

Another disadvantage of DDPG is sample inefficiency that occurs because of uninformed exploration. DDPG’s exploration policy selects a random action in every step with low probability to discover novel moves. This exploration policy makes succeeding in narrow passages highly unlikely, since these often require a very specific action sequence with no guiding reward signal along the way. Since the probability to randomly generate such a sequence of actions is extremely low, the agent can never reach the other side of the passage to collect the reward. Worse, randomly exploring this narrow corridor leads in practice to many collisions, making the agent avoid entering the passage in the first place.

In this work we propose to overcome the issue by providing the agent with motion planner demonstrations for failed work spaces. This is especially important in tight passages since instead of failing over-and-over, after the first failure the agent gets a useful signal that it can use to learn appropriate actions. Note that this strategy implicitly generates a curriculum: easier failure cases resolve earlier during training and the agent stops receiving demonstrations for those, while harder cases are still supplied with demonstrations. At the end of training we expect to succeed in most attempts and thus not use this exploration strategy, which is inline with other exploration strategies where the noise is being reduced over time, e.g using an $\epsilon$-greedy approach with a decreasing $\epsilon$ over time [47].

Let $\tau_{W}=(c^{\tau}_{0},a_{0},c^{\tau}_{1},a_{1}...a_{N-1},c^{\tau}_{N})$ be a trajectory produced by a motion planner, which is a solution to a failed workspace query $W=\{F,c_{0},g\}$, i.e $c^{\tau}_{0}=c_{0}$ and $FK(c^{\tau}_{N})=g$. In order to incorporate the information of $\tau_{W}$ to the replay buffer, we only need to assign a reward to every transition, to obtain $(c^{\tau}_{i},a_{i},r_{i},c^{\tau}_{i+1})$. Because $\tau_{W}$ is known to be a valid trajectory, this process is straightforward – by applying Eq. (6), to obtain $r_{i}$. This strategy can be seen as a ‘smart’ form of exploration, where instead of trying random actions, we are guided by the motion planner to try actions that result in success. We have found that this allows us to reduce the magnitude of the random exploration required for DDPG to work, thereby obtaining a much more stable algorithm.

VI Related Work

The idea of learning from experience to plan faster in motion planning has been studied under various approaches, such as using a library of past trajectories [7, 20, 6, 10], learning the sampling procedure [32, 19, 25], learning a latent representation of obstacles [19], and learning to select goals [12].

The idea of using a neural network for motion planning (termed here as NMP) dates back to the previous decade [15, 49]. Following recent advances in deep learning [24], interest in these methods has rekindled [35, 5, 50, 38, 39]. The potential advantages of NMP are the possibility of generalizing to different domains, working directly on raw perceptual signals such as images, and a low memory footprint: once trained, the NMP is a compact prediction model that does not require storing a large library of trajectories. While most previous work on NMP concerned the neural network architecture [e.g., 29, 39, 50, 5], in this work we focus on a parallel investigation – the training algorithm for NMP. We believe that with better architectures, our results can be further improved.

RL for continuous control has been explored extensively [42, 31]. Our method builds on two main ideas for improving RL: using expert demonstrations, and a model-based update for the policy gradient. While similar ideas have been explored in the literature [e.g., 34, 36, 17], our formulation is tailored for the NMP setting, and is, to the best of our knowledge, novel.

Using neural networks for planning in discrete tasks has been explored for navigation [48], task planning and combinatorial optimization [16, 9] and games [43]. Interestingly, the seminal AlphaGo program [43] exploited expert demonstrations and the known structure of the game to improve RL. Our work exploits similar properties of continuous motion planning domains, therefore requiring different methods.

VII Results

In this section we evaluate the DDPG-MP algorithm on various NMP domains, and show that it obtains state-of-the-art results in training an NMP model. In addition, we investigate the components that contribute to DDPG-MP’s success using an ablation study. Finally, we show that using DDPG-MP we can significantly improve upon the planning times of conventional motion planners. We investigate the following questions:

Is RL more suitable for NMP than imitation learning? 2. 2.

Can DDPG-MP obtain state-of-the-art results for NMP? 3. 3.

Can NMP models trained with DDPG-MP generalize to unseen work spaces with a high success rate? 4. 4.

Can we use NMP models to plan faster than sampling based motion planners?

We next describe the experimental settings we used.

NMP Architecture: As outlined above, in this work we wish to disentangle the questions of NMP architectures from the NMP training algorithm. Therefore, we chose popular neural network architectures that were the same for all methods we tested. Specifically, for domains without image inputs we used fully connected neural networks, which are popular for continuous control [13]. For domains with image inputs we used convolutional layers followed by fully connected layers [33, 31]. We provide full details of the network architecture in Section -B of the supplementary material.

Simulation environment: In our choice of a domain, we opted for an environment that can present challenging planning problems, yet be simple enough visually to not require complex perception efforts. We chose a 4DoF WidowX robotic arm [1], with movement restricted the XZ plane, and box-shaped obstacles in various sizes, positions, and orientations. Restricting movement to the plane allows us to capture the domain using a simple 2-dimensional image of the obstacles, which is natural for processing with convolutional neural networks, as discussed above. By varying the positioning of the obstacles, we can generate planning problems with various difficulty levels. We note that this setup is significantly more challenging than the point robots and car-like robots explored in previous NMP work [e.g., 39].

We used Tensorflow [2] for training the neural networks. Our code is available at https://github.com/tomjur/ModelBasedDDPG.

Evaluation metric: Evaluating the performance of agents trained with RL is a delicate matter [18], and in particular, how to choose the ‘best’ agent during training requires some validation metric. To address this, for each training run, the model iteration with highest test success-rate is taken and tested again on $1000$ new validation work-spaces. This asserts that the reported scores are not just a lucky sample, but accurately capture the success rate of the model (the validation success-rates are mostly lower than the best test success-rates).

Scenarios: Within the simulation described above, we conducted experiments on three scenarios, which we term simple, hard and vision, as depicted in Figure 3. The simple scenario (Figure 3(a)) contains just a single obstacle. The hard scenario (Figure 3(b)) was manually designed with a challenging obstacle configuration: three obstacles, such that the two top obstacles create a narrow passage where a sequence of very specific movements is required in order to cross. For the NMP problem, in both simple and hard scenarios the obstacle configuration is fixed, and the only variation is in the starting and goal positions. Formally, $W=(F,c_{0},g)$ contains a fixed $F$, and $c_{0}$ and $g$ are sampled from a distribution of feasible trajectories. The context vector $w$ is simply $g$.

The vision scenario, shown in Figure 3(c), is used to investigate generalization to unseen obstacle configurations. We generated a data set of $1000$ random obstacle configurations, partitioned to $80$% train and $20$% test. We report success rates on the test set. For this scenario all three parts of $W=(F,c_{0},g)$ are randomly sampled, and the context vector in this case contains both an image $I_{F}$ of the obstacle configuration, and the goal pose $g$.

We note that when sampling random work spaces (either obstacles or starting positions and goals), trivial problems, such as problems that do not require crossing obstacles, or where the goal is very close to the starting point, are frequent. To focus on interesting planning problems, we used rejection sampling to bias our testing to problems that require crossing from one side of an obstacle to the other. We note that in the hard and vision scenarios, the arm often needs to cross more than a single obstacle, making the task even more challenging. The difficulty of our domain is best appreciated in the accompanying video https://youtu.be/wHQ4Y4mBRb8. Full details of the sampling process are given in Section -B. We next present our results.

VII-A Imitation Learning vs. RL

As mentioned above, we have found that imitation learning does not provide accurate enough training for NMP. We demonstrate this here for the simple scenario, while similar results were obtained for other scenarios. We collected up to 160K expert demonstrations and trained imitation learning agents using behavioral cloning and DAgger. The results are presented in Figure 5, for various sizes of the training data set. The best behavioral-cloning and DAgger models reached a success rate of only $0.78$ and $0.8$ respectively. We also observed that the success-rate gains were diminishing, meaning that there is little contribution in adding more data. RL, on the other hand, was able to obtain a near-perfect NMP with a fraction of the expert demonstrations. This is explained in Figure 2 by observing the data distribution of RL and imitation learning, and noting that for RL the data is better distributed in the important areas – near obstacles.

VII-B RL for NMP

Next we evaluate DDPG-MP for NMP. We compare to two baselines: the original DDPG [31], and DDPG with HER [4], the current state-of-the-art in RL for learning goal-conditioned policies. Figures 4 and 6 plot the test success rate during training for the simple and hard scenarios. Tables II and III show the validation success rates for the hard and vision scenarios. Observe that DDPG-MP is the model with highest success rate in every experiment conducted, and on the hard scenario, alternative approaches only reach $0.31$ success rate, while DDPG-MP reached a near perfect success rate of $0.97$. These results demonstrate the effectiveness of DDPG-MP in learning neural network policies for motion planning, especially when the problem is challenging in the sense that it requires navigating through narrow passages.

In the following we perform an ablation study to identify the components that make DDPG-MP so successful.

VII-C Investigating the Model Based Actor Update

In this experiment, we demonstrate that only changing the actor objective from Eq. (3) to Eq. (7), greatly improves the precision and speed-of-convergence for the agent. To establish this, we compare DDPG with a variant of DDPG-MP that does not use the expert demonstration data, but uses instead vanilla DDPG exploration. As shown in figure 4 and table I, DDPG trains on the full 64K work-spaces and reaches a validation success rate of $0.825$. Meanwhile, DDPG-MP needs in average of $33082.66$ episodes to reach a test success score of $1.0$ and stop, achieving a validation of $0.9936$.

VII-D Investigating Exploration via Expert Demonstrations

The hard scenario is a challenging motion planning environment to learn: the narrow corridor requires a very precise sequence of actions without much room for errors. Moreover, since most work-space queries (which are presented randomly to the agent) do not include this portion of the state space, it is rare for the agent to see states in the corridor and is therefore prone to the problem of catastrophic forgetting. Under regular RL exploration strategies this task is either impossible or requires a large amount of episodes which the exploration happens to give the correct sequence over and over.

The targeted exploration method via expert demonstrations of Section V-C is shown to be an effective solution to this problem. In this section, we both measure the contribution of this strategy, and compare it to HER, a popular state-of-the-art exploration strategy.

We start by comparing the baselines DDPG with and without HER to DDPG-MP (full) as shown in Figure 6(a). We can see that DDPG-MP reaches a near-perfect validation success-rate of $0.9733$, while the DDPG and DDPG+HER only reach $0.318$ and $0.286$ each, demonstrating that the complete DDPG-MP algorithm is better in this scenario than both baselines.

Next, we would like to understand the contribution of using demonstrations to explore. For this purpose, in Figure 6(b) we compare DDPG-MP with a vanilla exploration strategy (i.e., replacing the expert demonstrations exploration with the standard DDPG exploration) to DDPG-MP with a HER exploration strategy (i.e., replacing the expert demonstrations with HER) and the complete DDPG-MP algorithm.

Figure 6 shows the success rate over the number of training episodes, and Table II shows the average validation success-rate, both clearly show that DDPG-MP (full) is the best model. Comparing DDPG-MP+HER (no expert) to DDPG-MP (no expert) we see that there is no benefit for using HER, and it even degrades the performance. We conclude that using HER is either not beneficial in this case, or requires more fine-tuning in order to make it work for these types of challenging motion planning scenarios. In Section -E of the supplementary material we provide an in-depth analysis of why HER is not well suited for such motion planning scenarios.

Finally, we analyze our proposed exploration strategy by comparing DDPG-MP (no expert) to DDPG-MP (full). DDPG-MP (full) reaches a near-perfect success rate of $0.9733$ compared to $0.81933$ of DDPG-MP (no expert), and also learns 4X faster. These results clearly show that DDPG-MP’s exploration strategy is beneficial.

VII-E Vision scenario: Generalization to Unseen Obstacle Configurations

The previous two experiments showed that DDPG-MP can generalize to unseen work spaces, where the obstacles are fixed but the start and goal configurations are varied. In this experiment we also investigate the effect of changing the obstacle configurations – can we train NMP models that generalize to previously unseen obstacle configurations? To answer this question we use the vision scenario, where with each work space, a visual input of the obstacles configuration is given to the model. We hypothesize that if unseen work spaces share some similarities with the work spaces seen during training, DDPG-MP will be able to generalize and solve the unseen scenarios with high accuracy.

Table III shows that all models reach high success rates, with DDPG-MP getting the highest validation success rate of $0.93$. We note that the individual scenarios that comprise the vision scenarios were selected from an easier distribution of obstacle configurations compared to the hard scenario, and this may be the reason that models such as DDPG and DDPG+HER are almost on-par with DDPG-MP.

VII-F Vision Scenario: Planning Times

Finally, we show that an NMP model trained with DDPG-MP has preferable running times to sample-based motion planners, for workspaces with an obstacle configuration not seen during training. To do this we compare the planning times it takes to produce trajectories for $100$ random workspaces with unseen obstacle configurations (from the vision scenario test set). DDPG-MP takes $8.55$ seconds to compute, while it takes $50.93$ seconds for OpenRAVE’s RRT motion planner – a 6X speedup in favor of DDPG-MP. The experiment was conducted on a Ubuntu 16 desktop machine, with a 12-core Intel i7-8700k 3.7GHz CPU, 32GB RAM and NVIDIA GeForceGTX 1080Ti. We note that OpenRAVE uses C++ code, while our code runs in non-optimized python. We expect that with dedicated optimization these results would improve.

These planning times should be contrasted with the accuracy rate of the NMP approach. For a $93$% success rate with a 6X speedup, a naive strategy that first tries to sample a plan using NMP, and if it fails (by running it through a collision checker) falls back to a conventional motion planner, a significant speedup on average would still be guaranteed. That said, we believe that smarter ways of combining the NMP with a conventional planner could be devised (e.g., along the lines of [16], which used an NMP as a search heuristic for discrete planning domains).

VIII Conclusion

In this work we presented DDPG-MP: an RL based algorithm for training a neural motion planner. We showed that our method significantly improves the training time and final accuracy of the learned planner, compared to both imitation learning and state-of-the-art RL approaches.

We further showed that our approach can be used to train neural networks that predict accurate motion plans given an image of the obstacles in the domain, and do so considerably faster than conventional motion planners.

In this work, we focused only on the learning algorithm for training a neural motion planner. Our results pave the way for further investigations of neural network architectures for motion planning, and combinations of learning based approaches with classical methods for motion planning.

-A DDPG-MP Algorithm

In this section we present the full DDPG-MP algorithm shown in Algorithm 1. Our extensions to DDPG can be found at:

Our actor update from Section V-B Equation (7) is found in line 4 of the ModelUpdate method. 2. 2.

The exploration via expert demonstration of Section V-C is shown on lines 6 and 7 of DDPG-MotionPlanner.

-B Experiments Settings and Parameters

In this section we describe the technical details we used for our scenarios in Section VII, as well as the hyper-parameters used for training the models.

We start by specifying the environment parameters used by our OpenRAVE [11] simulator. These values remain the same for all experiments for both the IL and RL agents. The simulator acts in joint movements the size of $0.025$ in the $L2$ norm (re-scaled the input action if required), while collision is tested on a grid spaced by $0.001$ intervals. Since it is unlikely that any agent would be able to reach the goal pose exactly, we allow that the finishing goal pose of the trajectory to have a distance smaller than $0.04$ between itself and the goal pose $g$.

We now discuss how to select work spaces that are suitable for training. A work space $W=(F,c_{0},g)$ must be challenging in order for NMPs trained on a corpus of such work spaces to have the ability to solve complex work spaces. In our work we decided that a work space can be inserted into our corpus of training data if exists an obstacle with center $o$ such that $o$ is between the start and goal poses, i.e $\|o-FK(c_{0})\|_{2}\leq\|g-FK(c_{0})\|_{2}$ and $\|o-g\|_{2}\leq\|g-FK(c_{0})\|_{2}$. This formulation only selects demonstrations that show movement from one side of an obstacle to another. We note that in the hard and vision domains, many work spaces require that the agent move across more than one obstacle.

When learning, each agent trains in an iterative process according to its structure and target functions, IL by epochs, and RL using noisy episode roll-outs. Between iterations, the model is constantly tested on work-spaces from the test set and the current test success-rate is evaluated. If the currently evaluated score is the best seen so far the model is saved as best iteration. After training, the best iteration model is re-evaluated using $1000$ new unseen work-spaces, and this is reported as the validation success-rate. We do this to make sure that the reported scores are not high due to randomness but actually reflect the performance of the agent. We note that the information gathered during testing and validation do not affect the training, namely these observed transitions are not added to any data-set the agents use for training.

The RL setting requires the following parameters: the future-rewards coefficient $\gamma$ is set to $0.99$. The reward at each time-step is combined from both $r^{T}_{t}$ as described in Eq. 6, and a domain reward. Our domain reward, penalizes the agent when moving a joint beyond the joint’s limit. The distance wasted is multiplied with a coefficient of $0.05$ and this is the additional penalty $r^{D}$.

All versions of DDPG and DDPG-MP use the following parameters: the agent collects $16$ episodes, after which $40$ model-update operations are executed sequentially (using different batches from the replay buffer). The learning continues until 64K, 80K and 160K episodes are seen for the simple, hard and vision scenarios respectively. The batch size is $512$, while the replay buffer size is $1000000$, the parameter $\tau$ which updates the target networks towards the online networks is set to $0.95$.

The exploration strategies’ parameters are as follows: for DDPG agents (with and without HER) and for DDPG (no expert) we use the same default random noise as in the original DDPG paper [31], completely random exploration occurs with $0.2$ probability, and [math]-mean Gaussian noise with $0.05$ std is added to every (training) action. Our model DDPG-MP which is not require extensive random exploration instead uses $0.02$ chance for a random action, and the Gaussian noise std is $0.005$. To refute the claim that the change of parameters is the reason our model learns better, we also ran tests where DDPG used this set of exploration parameters, and sure enough, DDPG discovered almost nothing and the success-rate remained close to zero for the entire training process.

Regarding the HER [4] parameters, we tried to take the default values, using the ”future” strategy, and $k=4$ as recommended. This worked on the hard scenario, however in the vision scenario, we needed to reduce $k$ to $1$, since the success rates of model with $k>1$ was dropping to zero after only a few training cycles without recovering.

DDPG-MP’s exploration policy utilizes motion plans for failed work-spaces. We found that instead of taking all the expert demonstrations that correspond to failed episodes, if we limit to a maximum of $8$ episodes per update (out of $16$) we get better performance.

In the next part we describe implementation details regarding the neural network models. All our networks start by concatenating the current joints $c_{t}$ to the information within the context vector $w$, and using it as input to subsequent layers. Meaning, in the simple and hard scenarios where there is no visual information, the result is just the goal pose $g$ concatenated with $c_{t}$. The vision scenario also contains visual information that first needs to be processed. We use a visual component (described below) that maps the input image $I_{F}$ describing the obstacles configuration, to a vector. This vector, gets concatenated to $c_{t}$ and the goal pose $g$.

The visual component is based on the DQN [33] design with less layers: we use two convulotional layers with $32$ and $64$ filters respectively, with kernel sizes of $8$ and $4$, and strides of $4$ and $2$. The result is flattened and goes through 2 fully-connected layers with $512$ hidden layers. All layers use the ReLU activation function [3]. The input for the visual component, $I_{F}$, is a $55\times 111$ grey-scale image.

The policy network (or actor) is a 4 hidden-layer NN with $200$ neurons in each layer and Elu activation units [8], with the final layer being a tanh activation. This is the same for all IL and RL agents. To make the network output a direction vector, the result of the tanh layer is normalized to always have a norm of 1. We found that in order to keep the learning stable, $L2$ regularization with coefficient of $1.0$ should be used on the input to the tanh function (the pre-activation values).

The critic has 3 layers with $400$ neurons each and the output of the third layer is concatenated the action input. Then 4 more layers with $400$ neurons each follow. The critic also uses Elu activations, except for the output that is linear, and we found that regularizing the critic weights with coefficient $10e-7$ works best.

As for the RL optimization, we used the Adam optimizer [23], and both actor and critic networks use a learn rate of $0.001$. We also found that using a gradient limit of $1.0$ was important to keep the gradients from exploding.

The imitation learning agents (both behavioral cloning and DAgger) use the same actor network structure as the RL agents. We train each agents for 100 epochs (regardless of the data-set size). These agents also use the Adam optimizer, but with a learning rate of $0.0001$ and a gradient limit of $5.0$.

All our agents were implemented using Tensorflow [2], which is a python deep-learning with auto-differentiation package.

-C Training IL Agents in the Simple scenario

In this section we elaborate on the preliminary tests regarding the usage of Imitation Learning in our settings. The goal of this experiment was to evaluate the success rate that IL agents can achieve and understand their data-set requirements, since in the motion planning settings obtaining motion-plans is costly. We found that while consuming a large amount of data, these agents do not reach the success rate expected from motion-planners. For more information regarding the reasons why IL is not well-suited see Section IV and Figure 2 in the main text.

Since the motion planning simulator described in Section -B requires a direction vector, we use the cosine-distance as a loss function (the cosine distance is measured between the predicted motion direction to the direction of the actual motion seen in training).

This experiment evaluates the success rate of both, Behavioral-Cloning and DAgger. For Behavioral-Cloning, even though we had 160K episodes of demonstrations, we also wanted to evaluate how the success improves as a function of the number of examples seen during training. Table IV shows that dependency, and we note that the it appears that Behavioral-Cloning agents reach a maximal validation success-rate of only about $0.79$, and that additional data does not improve the results.

DAgger [40], is an iterative method that uses an online expert to score states visited during training in order to avoid the distribution miss-match problem. We note, that in the motion planning case this approach is costly, since it requires the motion planner to plan an entire motion plan for every state seen during training (a single train episode contains hundreds of state visitations). This makes testing DAgger even more challenging since demonstrations are not computed in advance like in the IL case, but rather during the training process. We therefore relaxed this constraint and require the motion planner to plan for only the last 5 states in a failed work-space query.

Because of the expensive data requirements, we seed DAgger with 14K motion-plan demonstrations, and after every one of its 100 training epochs, DAgger gets 500 episodes that it can score using the policy mentioned above. Training for 100 epochs allows DAgger to see 64K episodes. With this configuration DAgger reaches a validation success rate of $0.8$, and we found that either adding more seeded motion-plans or online demonstrations does not affect the success rate. Meaning that similarly to Behavioral-Cloning the performance of DAgger is also capped albeit a bit higher.

-D Reward Models for Motion Planning: Design and Training

In this section we describe the structure of our reward model $\widetilde{r}$, the transition-termination model $\widetilde{p}$, as well how to train and test them. To utilize the full information available in a demonstration of a motion planning step $(w,c_{t},c_{t+1})$ we can exploit two labels: the transition class ($T_{free}$, $T_{goal}$ or $T_{col}$) and the reward value itself $r_{t}(w,c_{t},c_{t+1})$. This allows us to jointly train the models $\widetilde{r}$ and $\widetilde{p}$ in a single network.

We start by building and training a model for $\widetilde{p}$: this is a network with input $(w,c_{t},c_{t+1})$ (inline with our other models as described in section -B). The network outputs 3 hidden neurons that are the logits of the transition-class label: $l_{free}$, $l_{goal}$ and $l_{col}$ matching $T_{free}$, $T_{goal}$ or $T_{col}$. We then use the softmax function to get probabilities for each transition class, denoted by $\widetilde{p}_{free}$, $\widetilde{p}_{goal}$ and $\widetilde{p}_{col}$.

We train this network with a cross-entropy loss with the label being the true transition class label. The architecture of this network is simple: the input, is followed by 4 fully-connected hidden-layers with 100 neurons and Elu [8] activations each.

To compute $\widetilde{p}$, we can either take the $1-\widetilde{p}_{free}$ (probability of not ending in termination according to the softmax). Or we can first decide on the most probable between $l_{goal}$ and $l_{col}$ i.e $l_{termination}=\max(l_{goal},l_{col})$, and consider its softmax value between $l_{termination}$ with $l_{free}$. Empirically we found no difference between the two.

After this training process, we have a network that assigns probabilities to class transitions, we can train the predictor $\widetilde{r}$ for the reward. The input for this network is both (1) the same input as $\widetilde{p}$’s and (2) a 3 dimensional vector that is the probability for each transition class. The output for this network is a single scalar with a loss function of Mean-Squared-Error (MSE).We can use any network architecture to process this input, but in our experiments we simply used a linear classification of the inputs.

We notice that the above reward model allows us to use the output of the transition-labeling network i.e. $\widetilde{p}_{free}$, $\widetilde{p}_{goal}$ and $\widetilde{p}_{col}$ as an input to $\widetilde{r}$. This is what allows us to use this model in update rule in Equation (7) where the transition-label itself is not known in advance.

We train these model using the Adam optimizer [23] with a learning rate of $0.001$ a gradient limit of $5.0$ and a $L_{2}$ weight decay with coefficient $0.0001$. The batch size for the simple and hard scenarios is 10k, and for the vision scenario is 2K.

An important note about transition label imbalance: Empirically, in most motion planing scenarios, the number of transitions that are free $(w,c_{t},c_{t+1})\in T_{free}$ greatly outnumbers the other two classes. In all of our examples, transitions in $T_{free}$ make up of more than $0.98$ of the total transitions. This makes the other two classes very sparse, which make it hard to find a good solution (the trivial predictor that always predicts $T_{free}$ has an accuracy of more than $0.98$). To handle this issue we over-sampled the classes $T_{goal}$ and $T_{col}$ in each batch, making all classes equally proportional.

-E Analysis on HER Performance Degradation

In this section we attempt to give empirical evidence to explain why HER, is not a good exploration strategy to use on scenarios with narrow state passages. Our main observation is that HER has a tendency to amplify jittery movements.

An implicit assumption in HER is that every movement in an episode is a correct step towards the final state of that episode, however this assumption does not hold for work-spaces that contain unexplored narrow passages. Without a good exploration policy to guide the agent through the passage, the agent often becomes ”hesitant” to enter, and starts going back-and-forth near the entrance to the narrow region as it is trapped in a local minima, thus making no progress towards the goal on the other side of the passage. This phenomenon would only increase by HER, since the shaking actions are not only non-ideal progress even towards the final state, they are often even in the opposite direction.

Consider the results from Section VII-D of DDPG compared to DDPG+HER, the latter shows a consistent faster increase in test success-rate during the first 10K episodes of training, which aligns with the goal of HER to make rewards less sparse. However, as the agents start to consider the passages more and more as ”danger zones”, the jittering phenomena described above becomes more and more amplified, and DDPG surpasses DDPG+HER on average in the 25K episodes mark. We refer the reader to the uploaded movie which also shows the jittering phenomena with HER.