Precise but Natural Specification for Robot Tasks

TL;DR

This paper introduces Flipper, a natural language interface for robot task specification that combines formal planning language with natural language, visual feedback, and user-driven language extension to improve usability and expressiveness.

Contribution

It presents Flipper, a system that integrates formal task planning with natural language, visual feedback, and naturalization for flexible, precise robot task specification.

Findings

User study shows natural language interactions ease task description.

Users extend language with new concepts over time.

Flipper enables community sharing of language extensions.

Abstract

We present Flipper, a natural language interface for describing high-level task specifications for robots that are compiled into robot actions. Flipper starts with a formal core language for task planning that allows expressing rich temporal specifications and uses a semantic parser to provide a natural language interface. Flipper provides immediate visual feedback by executing an automatically constructed plan of the task in a graphical user interface. This allows the user to resolve potentially ambiguous interpretations. Flipper extends itself via naturalization: its users can add definitions for utterances, from which Flipper induces new rules and adds them to the core language, gradually growing a more and more natural task specification language. Flipper improves the naturalization by generalizing the definition provided by users. Unlike other task-specification systems, Flipper enables natural language interactions while maintaining the expressive power and formal precision of a programming language. We show through an initial user study that natural language interactions and generalization can considerably ease the description of tasks. Moreover, over time, users employ more and more concepts outside of the initial core language. Such extensions are available to the Flipper community, and users can use concepts that others have defined.

Full text

I Introduction

As robots move from controlled factory environments to homes, an important challenge is to allow end users to specify tasks for the robot in clear and unambiguous ways. While there are many programming languages for specifying tasks [1, 2, 3] and a number of tools that compile task specifications to robot action plans [4, 5, 6], their use is limited to users with programming experience and who have mastered the syntax and semantics of the particular language.

In order to make these languages accessible to end users, one can add “syntactic sugar” to a programming language. In its simplest form, users can express commands in a fixed structured subset of natural language [7]; utterances beyond this subset are rejected. Alternatively, one can use rich language models developed in the NLP community for arbitrary dialog understanding. However, the application to robot task planning is usually limited to a fixed set of the most common scenarios [8, 9, 10]; utterances beyond the pre-programmed tasks are rejected. Thus, users are either limited to a handful of restricted natual language idioms, or they can use their language freely, but the robot can only understand a few actions.

In this paper, we present Flipper, a system that aims to keep the precision of a programming language while allowing the ease of use of natural language. It does so by interactively extending the underlying grammar of a core formal language through induction of general rules from definitions. Flipper is able to adapt to the language style of its users while always maintaining a connection to the underlying programming language.

The core of Flipper is a high-level formal programming language for task specification and an executor that transforms that language into a robot action plan. A semantic parser [11] translates natural language utterances into this core language, allowing more flexibility in specifying tasks. While the core language is always available to programmers, Flipper further allows to extend the capability of the language and parsing through a process of naturalization [12]. Whenever the semantic parser fails to parse an utterance, the user can define the new utterance in terms of a sequence of utterances already understood (i.e., parsable) by the system. Flipper induces a set of new production rules in the core language from the user’s definition, thus extending the language. Furthermore, because the user’s definition might be too specific (i.e. work only for the current state of the world), Flipper uses semantic-preserving rewritings to generalize to an equivalent definition, matching the utterance and language model better.

Thus, over time, Flipper builds up a large lexicon of defined concepts through user interaction and generalization. Initial users program in the core language, but build up idioms natural to the domain; future users can use all concepts previously defined by the community. Along the way, each parsed form retains the precise semantics of the core language. In that way, Flipper helps bridge the gap between expressive and precise programming languages with restricted syntax on one side, and flexible but ambiguous natural language on the other.

How is Flipper’s naturalization mechanism different from defining concepts, for example using the library mechanism, in a programming language? Flipper exploits two key features of the robotics domain. First, it can provide quick visual feedback to the user about the effect of an utterance by simulating the plan on the world. This allows quick resolution of ambiguity. Second, unlike a library in a general purpose language in which a programmer parameterizes a function explicitly, Flipper uses grammar induction to create new parsing rules from definitions. Finally, it improves users’ definitions using an additional generalization mechanism based on rewritings. While grammar induction and rewriting may not be powerful enough for general-purpose computation, it works remarkably well in the restricted context of planning worlds.

Interacting with Flipper The user of Flipper instructs a single robot which moves in a world consisting of several connected rooms separated by walls, presenting obstacles for the robot. These rooms contain a number of items with different properties; for simplicity we consider here different colors and shapes. The robot can modify this world by moving to free locations, and by applying a set of actions, such as picking up or dropping items. Figure 1 shows a simulation of such a world (a robot symbol is a grey cube). The user specifies actions for the robot as an utterance. For example, the user might ask the robot to visit red triangle. If Flipper cannot understand an utterance, it will ask the user to define this utterance in terms of one or more commands that it already understands. In this example, the user can define the utterance in the core language: visit world containing item is red and is triangle. Based on that definition, Flipper induces new grammar rules which allows it to also understand related utterances such as “visit yellow circle.”

Once Flipper successfully parses an utterance into a task specification, it constructs a plan over the simulated world, and provides visual feedback first, showing what the robot would do. This is especially useful if there are multiple possible interpretations, e.g., due to ambiguities in the extended grammar. Once the user selects an interpretation, the robot proceeds with an action (ideally, in the real world; in our current implementation, still in the simulation). In case the plan is unrealizable, the user receives feedback specifying which primitive part of the action is not realizable.

Flipper integrates research performed in several different communities—NLP, planning, and program synthesis—in a non-trivial way. Overall, it provides an intuitive interface for end-user programming of robot tasks. The combination enables non-programmers to interact with robots in a flexible syntax, while assuming a number of power-users capable of grounding each utterance to a formal language. Specifically, we present the following contributions:

• •

an implementation of Flipper, a natural language interface for communicating with robots using naturalization, a process of interactive extending of a core programming language,

• •

improvement to the naturalization process using generalization by semantic-preserving rewriting,

• •

a preliminary study on 31 users showing that naturalization helps specify tasks in a more succinct way, that users define derived concepts as building blocks, and these derived concepts are reused by later users as their basis for interacting with the system.

II Related work

There are many systems commanding or communicating naturally with robots as well as creating programming languages for robots usable by non-programmers, starting with the seminal work in SHRDLU [13], and continuing over the years [10, 9, 14].

Formal and logical languages for planning have a long research tradition in AI and formal methods [1, 3, 2, 7]. Recently, there has been a focus on translating natural language into formal task specifications [7, 15, 8] to bridge the gap between end users and formal languages. These approaches either use a general-purpose NLP pipeline or a fixed set of “structured” templates. For example, [8] uses an NLP pipeline but restricts the set of actions a robot can perform to a fixed set of pre-programmed behaviors. [7] uses structured templates for a rich class of linear temporal logic specifications, but the user must only use the templates available. In contrast, Flipper allows both a natural style free of particular templates and a rich class of specifications. However, it assumes that the process of grammar extension is aided by users who are able to define natural language utterances in terms of previous utterances and ultimately the programming language.

The problem of grounding utterances to spatial relations between objects (but without naturalization) is tackled in [16, 17] by introducing a hierarchical structure that connects expressions such as beside the truck and beside the box. These concepts can be put in the context of reactive temporal commands [18]. It will be interesting to extend Flipper with reasoning about spatial relations in this way.

Flipper is inspired by and builds upon the work on naturalization of formal languages in Voxelurn [12], which considers a block world where a user can build various shapes of different colors. The application of naturalization to a robot world introduces new challenges: the language contains declarative and unrealizable commands and dynamic behavior that changes the state of the world. Additionally, Flipper improves naturalization by generalizing user-supplied definitions by program rewriting, which gives the power of defining new concepts to less programming-savvy users.

A similar approach of learning the language from users is presented in [19], but in the context of personal assistants. User’s feedback is employed in [20] to minimize the effort needed for additional annotation of data and iteratively improve their semantic parser that translates natural language utterances to SQL queries.

Beyond the robotics domain, an ensemble of a neural network and logistic regression models is used in [21, 22] to translate a task description from programming-help websites or IFTTT datasets into executable scripts. This line of work enables semantic parsing from less direct instructions, but is not easily adaptable to users interactively giving clues to the system about the meaning of the utterance. Also, these systems require lots of training data to function, whereas Flipper extends itself gradually.

III Background: Semantic Parsing with Naturalization

Flipper builds upon the idea of interactive semantic parsing, first presented in Voxelurn [12]. In this section we describe the main components of an interactive semantic parser, using examples from the Flipper domain.

Flipper is based on a core language whose syntax is defined in Figure 2. We shall describe the syntax in detail later; our simple examples should be understandable without a detailed understanding of the syntax.

We use semantic parsing [23] to parse utterances. Semantic parsing converts a natural language utterance into a ranked list of abstract syntax trees. The semantic parser outputs a ranked list because an utterance might be ambiguous, i.e. it could be parsed in several different ways. To rank the potential parses, the parser uses a statistical model $p_{\theta}(d|x,u)$ which assigns probabilities to derivations $d$, given utterance $x$ by user $u$ and using a set of parameters $\theta$. The features include whether the rule comes from the core language or is induced, whether the author of the rule is the same person as $u$, etc. The user visually inspects (in simulation) the three best-ranked derived programs. Based on the user’s choice, the model’s parameters $\theta$ are updated.

If the semantic parser is unable to parse an utterance, it asks the user to provide a definition for the utterance. A grammar induction module takes the definition for a particular utterance and creates one or more new rules in the underlying grammar for the language. The new rules extend the underlying language: in the future, the utterance as well as its variants become parsable.

We will use symbol $x$ for utterance, and $y$ for the provided definition. While $y$ must be fully parsable using the current grammar rules, only some parts of $x$ may be parsable. In order to induce new rules, the system identifies matches—parsable spans appearing in both $x$ and $y$. A set of non-overlapping matches is called a packing, and is the basis for generating new grammar rules. New grammar rules are introduced through simple packing, best packing, and alignment [12], which we describe below.

We illustrate simple and best packing on the following example. Suppose a user writes the (yet underfined) command $x:$ pick 3 items. Flipper asks for a definition and the user responds with $y:$ repeat 3 times pick item.

Simple packing considers pre-defined primitive categories for matching (such as colors, shapes or numbers). The only primitive match in the above example is the number 3, which in Flipper’s grammar has category Num. Therefore, a new rule is added to the grammar: Act $\to$ pick item Num $::=$ repeat Num times pick item.

Best packing considers maximal packings, i.e. those that would become overlapping by adding any other derivation, and chooses the packing that scores the best under the model $p_{\theta}$. The best scoring maximal packing for our example results in the rule Act $\to$ ItemAct Num items$::=$ repeat Num times ItemAct item. Note that this rule is more general than the one generated from the simple packing; with this rule in the grammar, Flipper will in the future understand commands such as drop 2 items. (While best packing is more general, it can potentially result in incorrect rules added to the grammar.)

Alignment is the third way to induce new rules and it considers the case when the utterance $x$ and the derivation $y$ align almost perfectly. As an example, for $x:$ throw item and $y:$ drop item, a new rule is added ItemAct $\to$ throw $::=$ drop.

IV Flipper

In this section we describe the Flipper language design, implementation and the generalization technique for user-supplied definitions. Flipper and its implementation are publicly available at flipper.mpi-sws.org.

IV-A Core Language

Flipper’s core language allows users to specify temporal tasks for a robot in an abstract world. Figure 2 shows the most interesting subset of the syntax of the core language. While the abstract world in which our robot operates may seem relatively simple, the core language allows expressing many interesting scenarios.

Intuitively, the core language interprets a program as a temporal goal for the robot in a grid world. The model of the grid world consists of a tuple $(M,I,r)$ where $M$ is a two-dimensional grid of points divided into free points and obstacles, $I$ is a set of items, and $r$ is the robot. Each item $i\in I$ has associated attributes such as color, shape, and a unique identifier. Additionally, if it is not held by the robot, it has a position which is a point in $M$. The robot $r$ has a position in $M$ and holds a possibly empty set of items.

The core language has a set of actions that manipulate the world and combines actions through standard imperative constructs such as sequencing, conditionals, or loops. We focus here on the non-standard parts of the language.

The propositions of the logical language specify either sets of items or sets of points, and can be combined using Boolean operations as usual. The syntactic class “Locations” describes sets of points in the grid $M$. The user can, for instance, start with the entire grid $M$ using the keyword world and filter the grid points to those which contain interesting items (e.g. one that is red) using the containing clause:

⬇

visit world containing item is red;

pick item is red

Similarly, the syntactic class “Items” describes sets of items. The items of interest can be selected by (possibly Boolean combinations of) their attributes. Once selected, all such items can be picked or dropped by using the every keyword

⬇

foreach point in world containing item is red

  { **visit** *p*oint; **pick** **every** *i*tem **is** *r*ed }

Actions modify the state of the world. For example, pick changes the world from a state where there is an item $i$ in the current position of the robot to a world in which the item is being carried by the robot. Actions can be temporal. The temporal action visit T, for a set $T\subseteq M$ of points, requires that the robot is moved to some position in $T$. The temporal action visit $T$ while avoiding $A$ additionally requires that along the way the robot never visits any point in $A\subseteq M$:

⬇

visit world containing item is red

  **while** **avoiding** *w*orld **containing** *i*tem **is** *c*ircle;

pick item is color red

This can be written in linear temporal logic (LTL) as $\Diamond T$ and $\lnot A\;\mathcal{U}\;T$, respectively.111 We remark that the temporal aspect of our core language is expressive enough to subsume LTL over finite traces with propositions ranging over robot’s locations (where visit while avoiding and if move constructs correspond to operators $\mathcal{U}$ and $\mathcal{X}$). We omit the formal encoding. In contrast to LTL-based robot planners [4, 15], Flipper’s core language is closer to imperative programming languages likely to be more familiar to programmers.

In a complex command, only some part of a command may be realizable. Consider

⬇

strict {while robot has item {drop item; move right}};

which instructs the robot to form, if possible, a horizontal line on the floor out of all the items the robot currently has (starting at robot’s current position and to the right). The robot may be able to move right once but not as many times as it has items. The default behavior is lenient in that it executes those parts of the command which are realizable and prints a warning about those which are not. The strict modifier allows to specify more rigid behavior that either performs the complete action or no part of it.

In a slight modification of the scenario above, if we want to place as many items to the right as possible:

⬇

drop item;

while possible {move right; drop item} {move right; drop item}

then we can use the while possible S T construct which repeatedly executes $T$ while $S$ is realizable.

IV-B Planning and User Interface

Given a goal expressed in the core language, Flipper generates an execution plan for the robot to satisfy it in the grid world. The planner is based on A* search [24] with Christofides algorithm [25] for iterated reachability. However, for a different language, one could use a different planner [15, 6, 26]. Fast planning is crucial for user interaction and we found A* to be fast and sufficient to find plans.

The generated plan is shown to the user visually in Flipper’s graphical interface (see Figure 1 and our supplemental video) by dynamic simulation of the robot’s moves in the UI. This visual feedback is important to verify that a possibly complex command given to the robot works as expected. Furthermore, ambiguities in the grammar can lead to several alternative plans being generated; the UI enables users to select the plan which best matches their intentions. Finally, users can view a list of all induced grammar rules in a sidebar, alongside contexts in which a rule was defined. They can additionally delete the rules defined by themselves.

IV-C Usage Examples

Naturalization not only helps accommodate many different language styles, it can also significantly simplify programs. Consider the task of distributing items of different colors to different rooms. Conceptually, this is fairly simple: the robot should first gather all red items and put them into room1, then do the same for the blue items and room2 and so on. Figure 3 shows an implementation of this specification in the core language. Each marked part corresponds to gathering items of a specific color and bringing them to a specific room. Clearly, there is a significant amount of redundancy.

With naturalization, we can accomplish the same task by first defining gather red as

⬇

foreach point in world containing item is red

{**visit** *p*oint; **pick** **every** *i*tem **is** *r*ed}

Then using this new command, we define red to room1 as

⬇

gather red; visit room1; drop every item is red

and finally we can accomplish our complete task with

⬇

red to room1; green to room2; blue to room3; yellow to room4

If we next want to put all items of different shapes to different rooms, the grammar induction allows us to re-use the commands defined above and simply write:

⬇

triangle to room1; circle to room2; square to room3

IV-D Generalization

We described how Flipper expands its formal language by means of inducing new rules from users’ definitions. A potential issue with this approach is that users sometimes provide definitions which are specific to the state of the presented world. Hence, the induced production rules will also be specific and less usable in other scenarios. For example, consider the situation from Figure 1 where a user wants to visit red triangle. The user may provide the definition $y=$ move right, since this command matches the user’s expectation for the given world. However, in a different world, the desired item may not be right next to the robot and hence the induced rule will not be correct.

Even if the user provides a more general definition, it may still not work in a different scenario. A user may define the utterance pick 3 items as $y=$ pick item; pick item; pick item. The naturalization method described in Section III cannot produce a meaningful production rule that would apply to commands of the form pick $n$ items, for arbitrary $n$.

We propose a solution in which Flipper synthesizes a number of definitions that could be used instead of only the one supplied by user. Let $w=(M,I,r)$ be the current state of the world. We call two definitions $d_{1}$ and $d_{2}$ $w$-equivalent if their execution trace on the world $w$ is the same. Since exploring the space of all possible $w$-equivalent definitions is not tractable, we limit the search to the following rewriting principles to produce a set $\mathcal{S}$ of $w$-equivalent definitions:

• •

transforming a sequence of identical actions into a loop

• •

transforming the robot’s movement actions into visiting a position defined by its coordinates or by a predicate over items present at the field

• •

transforming the robot’s picking or dropping actions into picking or dropping items defined by a predicate over items present at the field or at robot

Once the set $\mathcal{S}$ is obtained, Flipper uses only the definition $d\in\mathcal{S}\cup\{y\}$ that maximizes the score function $\sigma$ to induce new rules.

Example

In the world state depicted in Figure 1, assume the user gave a too specific definition $y=$ move right for the utterance visit red triangle. The set $\mathcal{S}$ contains (among others) the following alternative definitions: visit [4,0], visit world containing item, visit world containing item is red, visit world containing item is red and is triangle out of which the last one scores the best and is selected to induce new grammar rules. Hence, from now on, Flipper understands also e.g. visit blue circle.

The scoring function $\sigma$ captures how suitable a definition is to the induction of new production rules and how well it matches the original user’s intent expressed by the utterance. Therefore, we define the score function as $\sigma(d,x,u)=(p_{\theta}(d|x,u),\textit{\text{sim}}(d,u))$ and compare different values of $\sigma$ lexicographically. $p_{\theta}$ (described in Section III) captures the fit of the definition to the current semantic model. sim takes a bag-of-words representation of definition $d$ and utterance $u$. After eliminating stop-words, it calculates the cosine similarity between the averages of word-embeddings [27]. The implementation uses pre-trained word vectors of dimension 100 from GloVe [28].

Both the described generalization method and grammar induction described in Section III can create incorrect rules. In case there are multiple parses (presumably, some of them incorrect), Flipper offers a ranked list to the users. The user then chooses the correct one, which updates the parameters $\theta$, effectively making it less likely for incorrect rules to be applied in the future.

V Evaluation

We performed an initial user study to evaluate whether actual users use naturalization and thus whether Flipper makes programming robots easier. Hence, we focused the evaluation on the following:

usefulness of naturalization as a comparison between the workload of participants using Flipper and those using only the core language 2. 2.

usability of naturalization in terms of the number of new specifications users defined and used 3. 3.

naturalization across the group in terms of the number of specifications used which were defined by others (without knowing them in advance).

V-A Setup

We created a list of 21 tasks, ranging from easy (e.g. “get one green square”) to difficult (e.g. “bring all red items to a room that contains a yellow square”). The list of all tasks and all our experimental data is available at Flipper’s website. We recruited 31 participants, all with prior programming experience but without any knowledge of the system. We split them into three groups: group $A$ was only able to use the core language; group $B$ could additionally define new concepts; and group $C$ could on top of that use the concepts defined by an expert user (familiar with the system and the language) as well as by other participants from the group.

At the start of the experiment, each group had 30 minutes to familiarize themselves with Flipper and its core language by following a tutorial. Then the participants solved the 21 tasks as they saw fit, i.e., not necessarily in the most general way. The average time needed to solve all tasks was 90 minutes (but no deadline was set).

V-B Results

Usefulness and Usability of Naturalization

To assess the usefulness of naturalization, we measure for each participant the number of tokens used overall as well as in syntactically correct (parsable) commands coming either from the core or the induced language. The results are shown in Figure 4a. Considering only parsable commands, members of group B on average had to use 50% fewer tokens than the members of group A; and members of group C 58% fewer. This suggests that naturalization reduces users’ effort.

Then, for participants from groups B and C we compare the number of induced commands to the number of commands from the core language used to finish all tasks. The results in Figure 4b show that for groups $B$ and $C$, a significant fraction of used commands were induced. Two participants in group C, presumably with little programming experience, solved the tasks by explicit (step by step) instructions, with few definitions. This resulted in a very large number of core commands (183 and 112, respectively). Once these outliers were removed, we found that the average number of commands used was 33.5, ranging from 17 to 54.

Given that members of groups $B$ and $C$ were only motivated to use naturalization to ease their way through the tasks, we conclude that the naturalization technique was both useful and usable to them.

Naturalization across Users

Participants in group $C$ had access to the concepts defined by others. For this group, Figure 4c shows how many of the induced commands used were defined by the same participant or by others.

Participants were using a fair share of induced concepts, but still relied on the core language (on average, 54 core language commands and 51 induced language commands). When they used induced commands, they dominantly used commands defined by others rather than commands defined by themselves. Anecdotally, many participants used derived concepts without realizing they were derived.

Generalization (Section IV-D) improved definitions in several cases. For example, drop item; drop item; drop item was rewritten repeat 3 times drop item for the utterance drop 3 items and pick item is blue; pick item is blue into pick every item is blue for the utterance pick items is blue. This is a first step towards completely eliminating a requirement that a subset of users has to master the core language.

Types of Defined Concepts

Upon closer inspection of the concepts the participants defined, we see that a majority falls into two categories:

(1) simplifying individual commands and

(2) defining functions.

Examples for the first case are pick green square defined as pick item is green and is square and visit empty space defined as visit world minus {world containing item}.

For the second case a simple example is visit both triangle and green defined as

⬇

visit { {world containing item has shape triangle} and

{world containing item has color green} }

There were also function definitions that involved previous function definitions, such as line red being defined as fetch all red; while {robot has item} {drop item; move left}.

Conclusion

The outcome of the user study is twofold. First, it shows naturalization is beneficial in defining complex temporal tasks, but only with some programming experience, as it allows users to define commands that generalize and can be reused. Second, in order to serve non-programmers better, naturalization should be extended with orthogonal techniques which allow users to express intent through explicit examples which can be generalized directly.

VI Conclusion

We have shown that naturalizing a domain-specific programming language is well suited to provide a natural language interface to robot task specifications. Flipper provides the precision, expressivity, and extensibility of a programming language, while ensuring a natural experience for humans. Flipper adapts its language to its users by learning new concepts from them. The results of our initial evaluation are encouraging and suggest that a formal language for instructing robots can be turned with community effort into a domain specific natural language.