Prospection: Interpretable Plans From Language By Predicting the Future

TL;DR

This paper introduces a framework that enables robots to interpret natural language commands by generating interpretable plans with intermediate goals, using prospection to predict future consequences of actions for improved reasoning.

Contribution

It presents a novel approach that combines language understanding with predictive planning, allowing robots to generate interpretable, goal-oriented plans from natural language instructions.

Findings

Plans generated align well with human commands in simulated environments

The framework effectively predicts future consequences of actions

Demonstrates interpretability and fidelity of plans from natural language

Abstract

High-level human instructions often correspond to behaviors with multiple implicit steps. In order for robots to be useful in the real world, they must be able to to reason over both motions and intermediate goals implied by human instructions. In this work, we propose a framework for learning representations that convert from a natural-language command to a sequence of intermediate goals for execution on a robot. A key feature of this framework is prospection, training an agent not just to correctly execute the prescribed command, but to predict a horizon of consequences of an action before taking it. We demonstrate the fidelity of plans generated by our framework when interpreting real, crowd-sourced natural language commands for a robot in simulated scenes.

Tables

Table 1. TABLE I : L2 distances and accuracy when executing plans generated from either ambiguous natural language instructions or unambiguous template language.

	L2 distance in cm $\downarrow$					Success $\uparrow$
	Align	Grasp	Lift	Move	Release	Rate
Oracle	0.04	0.03	0.04	0.04	0.04	98.4%
GT Action	0.32	0.31	0.48	0.63	0.63	90.4%
Template	0.32	0.39	0.47	0.65	0.65	87.8%
Real Lang	0.51	1.23	1.50	2.39	2.40	77.1%

Table 2. TABLE II : Subgoal prediction accuracy ( ↑ ↑ \uparrow ) at different horizons with templated versus natural language. We see higher accuracy as we move closer to the end of the task, when the space of possible remaining plans is less ambiguous.

Horizon	$h=1$	$h=2$	$h=3$	$h=4$	$h=5$
Template
Verb	84.0%	87.9%	93.0%	97.4%	100.0%
To Object	91.5%	90.4%	93.0%	97.4%	100.0%
With Object	93.7%	91.8%	94.4%	96.8%	100.0%
Overall	82.9%	86.8%	92.5%	97.3%	100.0%
Real Lang
Verb	84.2%	87.8%	92.9%	97.2%	100.0%
To Object	87.8%	87.4%	89.8%	93.9%	98.2%
With Object	91.4%	90.1%	92.5%	96.4%	100.0%
Overall	79.6%	83.8%	89.1%	93.7%	98.2%

Table 3. TABLE A1 : Above are the initial and final visual frames for each task, next to the template language and human descriptions for examples from our training set. These examples illustrate why it can be so difficult for a model to predict specific motions that correspond to a particular natural language command, and further justify our approach for visualizing robot actions before execution. Specific reasons why each description is difficult to ground are indicated in bold

Before	After	Template	Human
		place yellow block on the red block	stack warm colors Unknown concepts
		stack the red block on the green one	move red right to same x and y axis as green Coordinate System
		place green on the yellow one	move the green box forward three spaces Spatial language
		stack the blue one on yellow	take the blue block in your hand and raise it above the table. move the block back and to the right until it is directly above the yellow block. lower the blue block down onto the yellow block and release it Latent details about hand movement
		put the yellow one on the green block	move the yellow cube to the right until it is on top of the green cube with the front half of the yellow cube touching the far half of the top of the green cube Denotes specific nuance

Equations

$$\omega ={\cos }^{-1}(2⟨{\hat{q}}_1,{\hat{q}}_2{⟩}^2).$$

$$C_{actor}(\hat{\theta },\theta )={\lambda }_{actor}\left\{\Vert \hat{p}-p{\Vert }_2^2+(1-⟨\hat{q},q⟩)+\Vert \hat{g}-g{\Vert }_2^2\right\}.$$

$$C(\hat{Z})=$$

$$\sum _{i\in h}\left(\genfrac{}{}{0ex}{}{{\lambda }_W\Vert {\hat{W}}_{t+i}-W_{t+i}{\Vert }_2^2}{+C_{actor}({\hat{\theta }}_{t+i},{\theta }_{t+i})+C_G({\hat{G}}_{t+i},G_{t+i})}\right).$$

Full text

I Introduction

A robot agent executing natural language commands must solve a series of problems. First, human language must be translated to an understanding of intent. For example, the command pick up the yellow block and place it on top of the red block corresponds to an intended change in world state that results in a yellow block on top of a red one. Given that understanding, an agent must plan a sequence of actions it can take to reach the target world state. In the above example, this could be (move(yellow); grasp(yellow); move(yellow, red); release(yellow)). Finally, these high level controls have to be executed in the world by servoing an arm to appropriate positions and controlling the gripper.

Each of these problems is challenging and has been investigated by existing research. Commonly, a pipeline approach is used, where each problem is addressed sequentially, and the outputs of one are fed to the next. In the example above, the semantic understanding that the goal is on(yellow, red) from natural language is passed to a high level controller. To simplify high level control prediction, robot perception is often augmented with visual semantics information, such as oracle object detections, bounding boxes, or 6d poses [1, 2]. In this work, we instead train a single model end-to-end that takes natural language and raw pixels, depths, and joint states to produce low-level controls to accomplish a goal.

Additionally, rather than treat the pipeline problems above independently, we introduce a prospection component to an agent’s training and inference, which facilitates “dreaming” about the consequences of chosen high level actions in the current scene. Prospection is the ability to reason about the consequences of future actions without executing them [3]. Our approach allows the agent to predict whether its high level actions will lead to undesirable world states.

We consider a simple pick-and-place task, where the goal is to stack one block on top of another (Figure 1). This is limited in that there are only a few high-level actions the robot can take in a given world. Still, it proves challenging when specified with real, crowd-sourced natural language.

In short, our contributions are:

• •

A language embedding rich enough to specify a sequence of actions to achieve a task-level goal.

• •

A DreamCell architecture to predict future world states from language and raw sensor observations.

• •

An approach to convert a task plan output from these DreamCells into low-level executions.

II Related Work

Our work draws inspiration from recent efforts on learning abstract representations for planning [4]. We build primarily on work in planning and natural language processing, with important future work in manipulation.

Communicating control and goals has traditionally been accomplished by specifying high level operations [5, 1, 6], via formal languages like the Problem Domain Description Language (PDDL) [7] or as a Hierarchical Task Network [8]. Such systems provide a straightforward way to compose black-box operations to solve problems. While we maintain an interpretable intermediate layer, our interface is natural language, most akin to [9, 10] though we work in a fully end-to-end differentiable paradigm where embedded language representations are learned alongside visual encodings of the world.

Our work is also motivated by the Universal Planning Networks which learn an embedding for images and a world state vector used to generate motion plans to goals specified via a target image [11]. That work learns a distance metric from the current state image to the target image which is used to perform rollouts for training and inference. While learned generic representations have notions of agency and planning, the produced plans lack human interpretability, which may be important to modularity and generalization [12]. Neural approaches and scaling robotic learning within simulation have become common as they allow for end-to-end training and can easily acquire more data (often from multiple domains) than otherwise possible on a physical device [2]. This has proven particularly important for RL-based approaches [13] and interpretability [14]. More generally there is a growing literature on learning deep representations that can be used to accomplish local control tasks or simple object manipulations [15, 16, 11, 17].

Core to our contribution is simulating the future actions and dynamics of our system. High level process simulation has been used in the NLP literature [18] without sensor data. Simultaneously, prospection has been used before in RL, often as a means of model-based control [19, 17, 20], and for fine control tasks like cutting [19]. In addition, our approach is compatible with work in Visual Robot Task Planning [4], which shows that prospective subgoal predictions can be used to generate task plans.

Complementary to our work is the growing literature in NLP focusing on complex grounding instructions with esoteric references and other long tail linguistic phenomena [21, 22]. Natural language communication with robotics also allows for learning joint multimodal representations [23, 24] which harness the unique perceptual and manipulation capabilities of robotics.

While accurate grasping and placement is not a focus of our work, it has been explored in the literature [25, 26, 27]. In particular, high-precision grasping with deep neural networks generally takes the form of predicting a grasp success classifier [25, 26].

III Problem Description

Given a natural language command and raw sensor readings for a scene, the task is to issue a sequence of low level controls to a robotic arm that accomplish the intended task.

Specification. A DreamCell takes in the initial $W_{0}$ and current $W_{t}$ simulation-based state observations and a natural-language sentence $s$ to produce a sequence of intermediate, latent-space goals $z_{i},\dots,z_{i+h}$ for this pick-and-place task up to horizon $h$. Each goal is a semantically meaningful break point in the execution, e.g., a completed grasp on the target.

DreamCells make three predictions at every time step:

A sequence of subgoals predictions, representing the next high level actions out to some planning horizon; 2. 2.

A sequence of hidden state representations $z_{i},\dots,z_{i+h}$ representing the results of these subgoals; and 3. 3.

The end effector command $\theta$ that parameterizes the low-level controller for task execution, consisting of a 6DOF pose and gripper command.

These predictions allow us to learn an interpretable, executable representation for hallucinating future world states.

Metrics. We measure both extrinsic performance on the task and intrinsic performance of DreamCell components. Extrinsically, we evaluate how well the robot agent completes the pick-and-place task. We record binary task success/failure as whether the target block to be moved is dropped within a threshold of its intended position based on the language command. We also record the average mean-squared error of the predicted end effector goal at each step of the task. This metric penalizes moving the wrong block while giving partial credit for moving the right block to towards the right place, even if it never arrives there or is placed unstably (e.g., if it falls off the target block after release). Intrinsically, we evaluate the language-to-action component by how closely the predicted sequence of subgoals matches ground truth execution.

IV Approach

We train the system end-to-end using simulated data. This allows us to automatically generate training sequences for both images and high-level subgoals. Subgoals take the form of semantic predicates like grasp and move with block arguments. To simplify notation, throughout the paper, we refer to the union of RGB, pose, and depth images with the single world state variable $W$. At every timestep the model is provided the current world observation ($W_{t}$), a description of the goal configuration in natural language (encoded as $\vec{L}$). In practice $W_{t}$ also includes the initial state $W_{0}$ to capture changes over time. All aspects of the model (including the encoders for both language and the world) are trained together. The model is trained on supervised demonstration data collected from an expert policy as per previous work [15, 4].

The basic unit in our model is the DreamCell (Fig. 2) which produces a sub-goal and a corresponding predicted image of the arm’s position at the next time step. This formulation allows for recurrent chaining of cells to rollout future goals and states (Fig. 1). Specifically, because the output of the DreamCell includes a deconvolved hallucination of the next world state ($\hat{W}_{t+1}$) we can simply continue to run the network forward, where true observations are replaced with the network’s predictions. Our cell has two outputs at every timestep $t$: 1. Subgoals ($G_{t}$) and 2. Predicted Worlds ($\hat{W}_{t+1}$). We provide intrinsic evaluations on future prediction performance in Section VI.

At inference time, we generate a task plan by rolling out multiple DreamCell timesteps into a possible future given state observations, $z$. The core of our approach is that these subgoals are converted into estimated world states $\hat{W}$ and associated end effector goals $\theta$, which are fed into a lower level controller $\pi$ that will convert them into trajectories.

IV-A DreamCell* Subgoal Module*

The subgoal module predicts the next subgoal from the current world state and the language instruction. It is formulated similarly to image captioning and sequence to sequence prediction. First, we use an LSTM [28] to encode the goal as expressed in language. Words are embedded as 64 dimensional vectors initialized randomly. We concatenate the final hidden state ($\vec{L}$) with the output of our world encoder ($z_{t}$) as the initial hidden state of a new LSTM cell for decoding.

We generate an output of $G_{t}=({\tt verb},{\tt to\_obj},{\tt with\_obj})$ tuples at each timestep, to a horizon of length five. We divide the subgoal $G_{t}$ into: ${\tt verb}$ the action to be taken, ${\tt to\_obj}$ the object to servo towards, and ${\tt with\_obj}$ the optional object in hand (e.g., ${\tt move(red,yellow)}$ moves the yellow block in hand to the target red block).

During training, we use cross entropy loss on all 5x3 generations. The first timestep in the RNN is passed the current hidden state and the At each timestep the RNN cell is passed the prediction from the previous timestep. As is common practice in the language modeling literature [29], we tie the emission and embedding matrix parameters. Traditionally, this is achieved by simply transposing a single matrix. Our model produces tuples by passing the hidden vector through three different feed-forward layers, so, to re-embed predictions we multiply by the three transposed embedding matrices and average the outputs to reconstruct an embedding.

IV-B DreamCell* World Predictor Module*

The prediction cell, shown in Fig. 3, takes in the current hidden state $z_{t}$ and the predicted subgoal $G_{t}$. Each prediction model outputs a predicted latent-state subgoal $\hat{z}_{t+1}$, such that $P(z_{t},G_{t})=\hat{z}_{t+1}$. In effect, we learn a many-to-one mapping across multiple timesteps, all of which need to produce the same goal. We should also be able to roll this simulation forward in time in order to visualize future actions. Training this prediction space is a difficult problem and requires a complex loss function involving multiple components.

The prediction cell is a simple autoencoder mapping inputs $W$ to and from a learned latent space, as show in Fig. 2. World observations $W_{t}$ and $W_{0}$ are combined into a single estimated latent state $z_{t}$. The vector containing the predicted subgoal $G_{t}$ is tiled onto this state. We use a bottleneck within each prediction cell to force information to propagate across the entire predicted image, and then estimate a change in latent state $\Delta z$ such that $\hat{z}_{t+1}=z_{t}+\Delta z$.

When visualizing the predicted image $\hat{W}_{t+1}$, we use a decoder consisting of a series of 5x5 convolutions and bilinear interpolation for upsampling.

IV-C Actor Module

The actor module, shown in Fig. 4, predicts the parameters of an action that can be executed on the robot. Specifically, it takes in $z_{t}$ and the current high-level action and predicts a destination end effector pose that corresponds to the robot’s position at $z_{t}$.

The architecture is a simple set of convolutions: the high-level action is concatenated with the current $z_{t}$ as in the prediction module, then a set of three 3x3 convolutions with 64, 128, and 256 filters, each followed by a 2x2 max pool. This is followed by a dropout and a single 512 dimensional fully connected layer, and then to $N_{verbs}\times 8$ outputs, predicting gripper command $g\in(0,1)$, Cartesian end effector position, and a unit quaternion for each high-level action verb. The gripper command uses a sigmoid activation where [math] represents closed and $1$ represents open, and Cartesian end effector position uses a tanh activation function. All pose values are normalized to be in $(-1,1)$.

A one-hot attention over action verbs chooses which pose and gripper command should be executed. In effect, the actor learns to compute a set of pose features for predicting the next manipulation goal and learns a simple perceptron model for each action verb in order to choose where the arm should go and whether the gripper should be opened or closed after the motion is complete.

IV-D Training

We train the encoder and decoder jointly when training the Prediction and Actor modules and optimize with Adam [30], using an initial learning rate of $1e-3$. We fix the latent state encoder and decoder functions after this step, then use the learned hidden space to train the Subgoal module.

Image Reconstruction Loss This determines how well our model can reconstruct an image from a given hidden state $z_{t}$, and is trained on the output of our visualization module. We used an L2 loss on pixels both for RGB and depth. Depth values were capped at 2 meters and were normalized to be between 0 and 1.

Subgoal Recovery Loss Image reconstruction losses are often insufficient for capturing fine details. This issue has motivated recent work on GANs [31]. These are often unstable, so we propose an alternative solution specialized to our problem. Since each successful high-level action has a predictable result, we jointly train a classifier that will recover the subgoal associated with each successive high-level action $\hat{G}_{t}$. We use $C_{G}(z_{t})$ as the classifier loss, minimizing cross entropy between the recovered estimate $\hat{G}_{t}$ and ground truth $G_{t}$.

Actor Loss Instead of estimating the full joint state of the robot as the result of a high-level action, our Actor module estimates the end-effector pose $\theta_{t}$ associated with sugboal $G_{t}$.

These poses are represented as $\theta=(\hat{p},\hat{q},\hat{g})$, where $\hat{p}$ is the Cartesian position, $\hat{q}$ is a quaternion describing the orientation, and $\hat{g}\in(0,1)$ is the gripper command. When regressing to poses, we use a mixture of the L2 loss between Cartesian position and a loss derived from the quaternion angular distance (to capture both spatial and rotational error). The angle between two quaternions $\hat{q}_{1}$ and $\hat{q}_{2}$ is given as:

[TABLE]

To avoid computing the inverse cosine as a part of the loss, we use a squared distance metric. In addition, normalize gripper commands to be between 0 and 1, where 0 is closed and 1 is open, and trained with an additional L2 loss on predicted gripper commands. Given estimated pose $\hat{\theta}$ and final pose $\theta$, we calculate pose estimation loss:

[TABLE]

Object Pose Estimation Loss It is important to ensure that our learned latent states $z_{t}$ capture all the necessary information to perform the task. As such, we use an augmented loss $C_{obj}(z_{t})$ that predicts the position of each of the four blocks in the scene at the observed frame. This information is not used at test time, but is structurally identical to the pose estimation loss $C_{pose}$

Combined Prediction Loss The final loss function for predicting the effects of performing a sequence of high-level actions is then:

[TABLE]

IV-E Execution

When executing in a new environment, the robot agent takes in the current world state $W_{0}$ and a natural language instruction $L$. The agent computes a future prediction using a DreamCell, by rolling out predicted goals $G_{t},\dots,G_{t+H}$ which generate latent space subgoals. These subgoals can then be visualized to provide insight into how the robot expects the task to progress. This also illuminates misunderstandings and limitations of the system (see Analysis VI).

The system generates new prospective plans out to a given planning horizon. After predicting the next subgoal $z_{t+1}$, it will then use the actor to estimate the next motion goal $\theta_{t+1}$. This goal is sent to the low-level execution system, which in our case is a traditional motion planner that does not have knowledge of object positions. In our case, the planner used was RRT-connect [32], via MoveIt

In the future, these subgoals shown to the user, who can give the final confirmation on whether or not to execute this hallucinated task plan if it accomplishes what they requested. Alternatively, the user could input a new $L$, or the agent could sample a new sequence of goals.

V Experiment Setup

We performed a number of variations on a simple block stacking task. All experiments were performed in simulation. We collected 5015 trials using a sub-optimal expert policy, of which 2370 were successes. Our model was trained only on successful examples.

We generated trials using a simple simulation of an ABB YuMi robot picking up 3.5 cm cubes. There were four cubes, one of each color: red, green, yellow, blue. When collecting data, we first randomly compute a table position within a 50 cm box centered in front of the robot. Blocks were randomly placed in non-intersecting positions on this table. The arm’s initial position was also randomized to an area off the right side of the table.

We selected manipulation goals at random, and provided a simple expert policy which moved to pick up each object using an RRT motion planner. The plan has five steps: align with the top of a random block, grasp that block and close the gripper, lift the block off the table, move the block to atop another block, and then release the block.

We collected natural language commands from human annotators through the Mechanical Turk crowd-sourcing platform.111https://www.mturk.com/ Annotators were shown two scene images: one before and one after a block had been stacked on another block. They were instructed to give two distinct commands that would let someone create the second scene from the first (Figure 5), and were paid $0.25 per such annotation. For each of our 2370 successful trials, we obtained two language commands describing the high level pick-and-place goal. On average, commands are 11 words long.222Participants used 389 unique words after lowercasing and tokenization. We compare Turk data to unambiguous templated language that was procedurally generated from the manipulated blocks.

VI Results

We ran a set of experiments on our simulation, and computed task execution success rate. We analyze the performance of the Subgoal and Predictor modules given different classes of language input.

VI-A Execution Results

Finally, we test our model on a set of held-out scenarios, and compare to ground-truth execution. We compared accuracy of the estimated motion under each of three conditions: with oracle subgoals $G_{t}$ from the test data, with unambiguous templated language, and with natural language. Position accuracy results are shown in Table I.

In all cases, we compute an execution plan $\hat{G}_{1},\dots,\hat{G}_{5}$ at the beginning and use our Predictor and Actor networks to follow this execution plan until all steps have been executed. Results are shown in Table I. We count successes when the block was moved to within 1.5 cm of the target in the x and y direction, and 0.5 cm z of the final position from which it was dropped.

We see only a handful of failures when the robot was sent to ground truth “oracle” poses, due to stochastic interactions between the objects and gripper and randomness at the control level. $88.1$% of execution with ground truth actions, $88.0$% of unambiguous templated language, and $84.0$% of natural language were successfully able to pick up a block and put it in the right area—indicating a high level of precision independent of the task specification. Grasp success rates tended to be very high. The most dramatic failures we observed were situations where one or more necessary blocks was out of the camera’s viewpoint, in which case our vision-based system fails by default.

The similar performance between templated language and ground truth actions suggests that unambiguous, templated language is insufficient to demonstrate the language learning capabilities of our system. We find our method is robust to real natural language from Mechanical Turk workers, achieving 95% of the success rate seen on unambiguous templates.

VI-B Subgoal Module

We analyze the language learning component of our model. A full breakdown of subgoal prediction accuracy is given in Table II. Performance was comparable between templated language and natural language data collected from Amazon Mechanical Turk. We see that it is more difficult to make accurate predictions on real language data. Additionally, accuracy is remarkably consistent over time, meaning that the model properly learned the correct sequence of actions that should be executed. Accuracy farther out into the future is stable because the network knows when and how a sequence should end.

There are two major sources of error we observe in these examples. First, the difference in accuracy between Mechanical Turk language and templated language is largely explained by the ambiguity and underspecificity in Turk commands (e.g., not specifying a destination after a grasp). Second, the overall error is largely due to sequence error at transition points, where multiple possible actions are reasonable depending on whether or not the low level control has arrived at its destination. This further supports our hypothesis that we need to reason about all three sub-problems jointly.

VI-C Prediction Model

The role of the prediction model is to generate subgoal predictions $\hat{z}_{t+1},\dots,\hat{z}_{t+h}$ representing the $h$ next actions that the robot can take in order to perform the task. Fig. 6 shows an example of one course of error we see during these prediction rollouts. The top two rows show a sequence of predictions coming from the sequence to sequence model, while the bottom row shows predictions using ground truth actions from our data. In this case, we see that the sequence to sequence model started the grasp verb earlier than the ground truth execution, but both models generate good image predictions.

As we can see in Table I, there is persistently some error in the low-level predictions from our actor module, even when given oracle arguments Average placement error increases as we move away from the ground-truth arguments. Often failures occur because the object is not clearly visible in the first frame.

Our reconstruction results have another advantage, however, as seen in Fig. 6: they are clearly interpretable, which means that the robot can readily justify its decisions even when it does make a mistake, facilitating a human user providing a new instruction that considers this mistake. Overall, these results show that we can learn representations for a task that are sufficient for planning and execution purely from language and raw sensory data.

We performed an ablation analysis on the best prediction models to determine how much they use information from different layers. In particular, we see similar performance when training without the image loss ($89.5\%$ successful on held out test data) and without the image and object losses ($88.6\%$ successful). This suggests that our image reconstruction loss may help, and certainly does not have a negative impact.

VII Conclusions

We present an approach for inferring interpretable plans from natural language and raw sensor input using prospection. Our DreamCell architecture predicts future world states from language and raw sensor observations, facilitating high level plan inference that can be converted into low-level execution. Prospection enables end-to-end plan inference that is agnostic to the nature of sensory input and low-level controller modules. In the future, using this architecture to bootstrap language understanding for execution on a real robot using sim-to-real transfer techniques could facilitate end-to-end control on a physical platform.

VIII Acknowledgements

This work was funded in part by the National Science Foundation under contract no. NSF-NRI-1637479, and the DARPA CwC program through ARO (W911NF-15-1-0543). We would like to thank Jonathan Tremblay for valuable discussions.

A-i Templated vs Real Language

Table A1 contains examples of the different types of language that occur when asking humans to describe the action versus using templated language.

A-ii Prediction Results

One advantage of proposed DreamCell system is that it allows us to generate multiple hallucinations of possible futures. Here, we show example plans generated from four unseen test environments, given a natural-language prompt. We show predictions for the first four high level actions: align, grasp, lift, and move_to. Environments and trials were chosen at random, and should be indicative of performance on the prospection problem.