Improving Grounded Natural Language Understanding through Human-Robot Dialog

TL;DR

This paper presents an end-to-end system enabling robots to understand and adapt to natural language commands through dialog, improving grounding and parsing in both virtual and real-world environments.

Contribution

It introduces a dynamic learning pipeline that uses clarification dialogs to enhance language understanding and concept grounding in robotic systems.

Findings

Successful transfer from virtual to physical robot platform

Improved language parsing accuracy through dialog-based learning

Effective grounding of perceptual concepts in real-world tasks

Abstract

Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically---continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world.

Figures

Tables

Table 1. TABLE I : Samples of the agent’s static dialog policy π 𝜋 \pi for mapping belief states to questions.

$B$ max per role	Min
(action, patient,	Prob	Question	Type
recipient, source, goal)	$B$ Role
$(\varnothing ,\varnothing ,\varnothing ,\varnothing ,\varnothing )$	All	What should I do?	Clarification
$(\text{walk},\varnothing ,\varnothing ,\varnothing ,r_1)$	action	You want me to go somewhere?	Confirmation
$(\text{deliver},\varnothing ,p_1,\varnothing ,\varnothing )$	patient	What should I deliver to $p_1$?	Clarification
$(\text{relocate},\varnothing ,\varnothing ,\varnothing ,\varnothing )$	source	Where should I move something from on its way somewhere else?	Clarification
$(\text{relocate},o_1,\varnothing ,r_1,r_2)$	-	You want me to move $o_1$ from $r_1$ to $r_2$?	Confirmation

Table 2. TABLE II : The average number of clarification questions agents asked among dialogs that reached the correct task. Also given are the p 𝑝 p -values of a Welch’s t 𝑡 t -test between the Tr ained ∗ ( Perception ) and Tr ained ( Parsing+Perception ) model ratings against the In itial model ratings.

Agent	Clarification Questions $\downarrow$
	Navigation $\left(p\right)$	Delivery $\left(p\right)$	Relocation $\left(p\right)$
In	$3.02\pm 6.48$$\left(.00\right)$	$6.81\pm 8.69$$\left(.00\right)$	$22.3\pm 9.15$$\left(.00\right)$
Tr∗	$4.05\pm 8.81\left(.46\right)$	$8.16\pm 13.8\left(.53\right)$	$23.5\pm 6.07\left(.67\right)$
Tr	$1.35\pm 4.44\left(.11\right)$	$7.50\pm 9.93\left(.72\right)$	$19.6\pm 7.89\left(.47\right)$

Table 3. TABLE III : The average Likert rating given on usability survey prompts for each task across the agents. Bold indicates an average Tr ained ∗ ( Perception ) and Tr ained ( Parsing+Perception ) model ratings significantly higher than the In itial model ( p < 0.05 𝑝 0.05 p<0.05 ) under a Welch’s t 𝑡 t -test.

Agent	Usability Survey (Likert 1-7) $\uparrow$
	Navigation $\left(p\right)$	Delivery $\left(p\right)$	Relocation $\left(p\right)$
In	$3.09\pm 2.04$$\left(.00\right)$	$3.20\pm 2.12$$\left(.00\right)$	$3.37\pm 2.17$$\left(.00\right)$
Tr∗	$3.51\pm 2.05\left(.09\right)$	$3.60\pm 2.09\left(.12\right)$	$3.60\pm 2.08\left(.37\right)$
Tr	$3.76\pm 2.07\left(.01\right)$	$3.87\pm 2.10\left(.01\right)$	$3.93\pm 2.16\left(.04\right)$

Equations

$$\mathcal{B}(r,a)\leftarrow (1-\rho )\mathcal{B}(r,a)+\rho {\mathcal{B}}_x(r,a),$$

Full text

I Introduction

As robots become ubiquitous across diverse human environments such as homes, factory floors, and hospitals, the need for effective human-robot communication grows. These spaces involve domain-specific words and affordances, e.g., turn on the kitchen lights, move the pallet six feet to the north, and notify me if the patient’s condition changes. Thus, pre-programming robots’ language understanding can require costly domain- and platform-specific engineering. In this paper, we propose and evaluate a robot agent that leverages conversations with humans to expand an initially low-resource, hand-crafted language understanding pipeline to reach better common ground with its human partners.

We combine bootstrapping better semantic parsing through signal from clarification dialogs [1], previously using no sensory representation of objects, with an active learning approach for acquiring such concepts [2], previously restricted to object identification tasks. Thus, our system is able to execute natural language commands like Move a rattling container from the lounge by the conference room to Bob’s office (Figure 5) that contain compositional language (e.g., lounge by the conference room understood by the semantic parser and objects to be identified by their physical properties (e.g., rattling container). The system is initialized with a small seed of natural language data for semantic parsing, and no initial labels tying concept words to physical objects, instead learning parsing and grounding as needed through human-robot dialog.

Our contributions are: 1) a dialog strategy to improve language understanding given only a small amount of initial in-domain training data; 2) dialog questions to acquire perceptual concepts in situ rather than from pre-labeled data or past interactions alone (Figure 1); and 3) a deployment of our dialog agent on a full stack, physical robot platform.

We evaluate this agent’s learning capabilities and usability on Mechanical Turk, asking human users to instruct the agent through dialog to perform three tasks: navigation (Go to the lounge by the kitchen), delivery (Bring a red can to Bob), and relocation (Move an empty jar from the lounge by the kitchen to Alice’s office). We find that the agent receives higher qualitative ratings after training on information extracted from previous conversations. We then transfer the trained agent to a physical robot to demonstrate its continual learning process in a live human-robot dialog.111A demonstration video can be viewed at

https://youtu.be/PbOfteZ_CJc?t=5.

II Related Work

Research on the topic of humans instructing robots spans natural language understanding, vision, and robotics. Recent methods perform semantic parsing using sequence-to-sequence [3, 4, 5] or sequence-to-tree [6] neural networks, but these require hundreds to thousands of examples. In human-robot dialog, gathering information at scale for a given environment and platform is unrealistic, since each data point comes from a human user having a dialog interaction in the same space as a robot. Thus, our methods assume only a small amount of seed data.

Semantic parsing has been used as a language understanding step in tasks involving unconstrained natural language instruction, where a robot must navigate an unseen environment [7, 8, 9, 10, 11], to generate language requests regarding a shared environment [12], and to tie language to planning [13, 14, 15, 16]. Other work memorizes new semantic referents in a dialog, like this is my snack [17], but does not learn a general concept for snack. In this work, our agent can learn new referring expressions and novel perceptual concepts on the fly through dialog.

Mapping from a referring expression such as the red cup to a referent in the world is an example of the symbol grounding problem [18]. Grounded language learning bridges machine representations with natural language. Most work in this space has relied solely on visual perception  [19, 20, 21, 22, 23, 24], though some work explores grounding using audio, haptic, and tactile signals produced when interacting with an object [25, 26, 27, 28]. In this work, we explicitly model perceptual predicates that refer to visual (e.g., red), audio (e.g., rattling), and haptic properties (e.g., full) of a fixed set of objects. We gather data for this kind of perceptual grounding using interaction with humans, following previous work on learning to ground object attributes and names through dialog [29, 30, 31, 32, 33]. We take the additional step of using these concepts to accomplish a room-to-room, pick-and-place task instructed via a human-robot dialog. To our knowledge, there is no existing end-to-end, grounded dialog agent with multi-modal perception against which to compare, and we instead ablate our model during evaluation.

III Conversational Agent

We present a end-to-end pipeline (Figure 2) for an task-driven dialog agent that fulfills requests in natural language.222The source code for this dialog agent, as well as the deployments described in the following section, can be found at

https://github.com/thomason-jesse/grounded_dialog_agent.

III-A Semantic Parser

The semantic parsing component takes in a sequence of words and infers a semantic meaning representation of the task. For example, a relocate task moves an item (patient) from one place (source) to another (goal) (Figure 2). The agent uses the Combinatory Categorial Grammar (CCG) formalism [34] to facilitate parsing.

Word embeddings [35] augment the lexicon at test time to recover from out-of-vocabulary words, an idea similar in spirit to previous work [36], but taken a step further via formal integration into the agent’s parsing pipeline. This allows, for example, the agent to use the meaning of known word take for unseen word grab at inference time.

III-B Language Grounding

The grounding component takes in a semantic meaning representation and infers denotations and associated confidence values (Figure 2). The same semantic meaning can ground differently depending on the environment. For example, the office by the kitchen refers to a physical location, but that location depends on the building.

Perceptual concepts like red and heavy require considering sensory perception of physical objects. The agent builds multi-modal feature representations of objects by exploring them with a fixed set of behaviors. In particular, before our experiments, a robot performed a grasp, lift, lower, drop, press, and push behavior on every object, recording audio information from an onboard microphone and haptic information from force sensors in its arm. That robot also looked at each object with an RGB camera to get a visual representation. Summary audio, haptic, and visual features are created for each applicable behavior (e.g., drop-audio, look-vision), and these features represent objects at training and inference time both in simulation and the real world.333That is, at inference time, while all objects have been explored, the language concepts that apply to them (e.g., heavy) must be inferred from their feature representations.

Feature representations of objects are connected to language labels by learning discriminative classifiers for each concept using the methods described in previous work [37, 31]. In short, each concept is represented as an ensemble of classifiers over behavior-modality spaces weighted according to accuracy on available data (so yellow weighs look-vision highly, while rattle weighs drop-audio highly). While the objects have already been explored (i.e., they have feature representations), language labels must be gathered on the fly from human users to connect these features to words.

Different predicates afford the agent different certainties. Map-based facts such as room types (office) can be grounded with full confidence. For words like red, perceptual concept models give both a decision and a confidence value in $[0,1]$. Since there are multiple possible groundings for ambiguous utterances like the office, and varied confidences for perceptual concept models on different objects, we associate a confidence distribution with the possible groundings for a semantic parse (Figure 2).

III-C Clarification Dialog

We denote a dialog agent with $\mathcal{A}$. Dialog begins with a human user commanding the agent to perform a task, e.g., grab the silver can for alice. The agent maintains a belief state modeling the unobserved true task in the user’s mind, and uses the language signals from the user to infer that task. The command is processed by the semantic parsing and grounding components to obtain pairs of denotations and their confidence values. Using these pairs, the agent’s belief state is updated, and it engages in a clarification dialog to refine that belief (Figure 2).

The belief state, $\mathcal{B}$, is a mapping from semantic roles (components of the task) to probability distributions over the known constants that can fill those roles (action, patient, recipient, source, and goal). The belief state models uncertainties from both the semantic parsing (e.g., prepositional ambiguity in “pod by the office to the north”; is the pod or the office north?) and language grounding (e.g., noisy concept models) steps of language understanding.

The belief states for all roles are initialized to uniform probabilities over constants.444Half the mass of non-action roles is initialized on the $\varnothing$ constant, a prior indicating that the role is not relevant for the not-yet-specified action. We denote the beliefs from a single utterance, $x$, as $\mathcal{B}_{x}$, itself a mapping from semantic roles to the distribution over constants that can fill them. The agent’s belief is updated with

[TABLE]

for every semantic role $r$ and every constant $a$. The parameter $\rho\in[0,1]$ controls how much to weight the new information against the current belief.555We set $\rho=0.5$ for clarification updates.

After a belief update from a user response, the highest-probability constants for every semantic role in the current belief state $\mathcal{B}$ are used to select a question that the agent expects will maximize information gain. Table I gives some examples of the policy $\pi$.

For updates based on confirmation question responses, the confirmed $\mathcal{B}_{x}$ constant(s) receive the whole probability mass for their roles (i.e., $\rho=1$). If a user denies a confirmation, $\mathcal{B}_{x}$ is constructed with the constants in the denied question given zero probability for their roles, and other constants given uniform probability (so Equation 1 reduces the belief only for denied constants). A conversation concludes when the user has confirmed every semantic role.

III-D Learning from Conversations

The agent improves its semantic parser by inducing training data over finished conversations. Perceptual concept models are augmented on the fly from questions asked to a user, and are then aggregated across users in batch.

Semantic Parser Learning From Conversations

The agent identifies utterance-denotation pairs in conversations by pairing the user’s initial command with the final confirmed action, and answers to questions about each role with the confirmed role (e.g., robert’s office as the goal location $r_{1}$), similar to prior work [1]. Going beyond prior work, the agent then finds the latent parse for the pair: a beam of parses is created for the utterance, and these are grounded to discover those that match the target denotation. The agent then retrains its parser given these likely, latent utterance-semantic parse pairs as additional, weakly-supervised examples of how natural language maps to semantic slots in the domain.

Opportunistic Active Learning

Some unseen words are perceptual concepts. If one of the neighboring words of unknown word $x_{i}$ is associated with a semantic form involving a perceptual concept, the agent asks: I haven’t heard the word ‘$x_{i}$’ before. Does it refer to properties of things, like a color, shape, or weight? If confirmed, the agent ranks the nearest neighbors of $x_{i}$ by distance and sequentially asks the user whether the next nearest neighbor $t_{p}$ is a synonym of $x_{i}$. If so, new lexical entries are created to allow $x_{i}$ to function like $t_{p}$, including sharing an underlying concept model (e.g., in our experiments, tall was identified as a synonym of the already-known word long). Otherwise, a new concept model is created for $x_{i}$ (e.g., in our experiments, the concept red).

We introduce opportunistic active learning questions [2] as a sub-dialog, in which the agent can query about training objects local to the human and the robot (Figure 5). This facilitates on the fly acquisition of new concepts, because the agent can ask the user about nearby objects, then apply the learned concept to remote test objects (Section IV-C).

IV Experiments

We hypothesize that the learning capabilities of our agent will improve its language understanding and usability. We also hypothesize that the agent trained in a simplified world simulation on Mechanical Turk can be deployed on a physical robot, and can learn non-visual concepts (e.g., rattling) on the fly that could not be acquired in simulation.

IV-A Experiment Design

The agent (and corresponding robot) can perform three high-level tasks: navigation (the agent goes to a location), delivery (the agent takes an object to a person), and relocation (the agent takes an object from a source location to a goal location). We denote 8 (randomly selected) of the 32 objects explored in prior work [38] as test objects and the remaining 24 as training objects available for active learning queries. We randomly split the set of possible task instantiations (by room, person, and object arguments) into initialization (10%), train (70%), and test sets (20%).

Initialization Phase

Sixteen users (graduate students) were shown one of each type of task (from the initialization set) and gave two high-level natural language commands for each (initial and rephrasing). We used a subset of these utterances666Commands that would introduce rare predicates were dropped. as a scaffold on which to build a seed language-understanding pipeline: an initial lexicon and a set of 44 utterance-semantic parse pairs, $D_{0}$.777An experimenter performed the annotations to create these resources in about two hours.

Training Procedure

The initial pipeline is used by a baseline agent $\mathcal{A}_{1}$; we denote its parser $\mathcal{P}_{1}$ trained on $D_{0}$, and denote untrained concept models for several predicates $P_{c,1}$. That is, the initial lexicon contains several concept words (like yellow), but no labels between objects and these concepts. All learning for the parsing and perception modules arises from human-agent conversations.

We divide the training procedure into three phases, each associated with 8 different objects from the active training set of 24. The perceptual concept models are retrained on the fly during conversations as questions are asked (e.g., as in Figure 1). The parsing model is retrained between phases. Each phase $i$ is carried out by agent $\mathcal{A}_{i}$, and training on all phase conversations yields agent $\mathcal{A}_{i+1}$ using concept models $\mathcal{P}_{i+1}$ and parser $P_{c,i+1}$. In each phase of training, and when evaluating agents in different conditions, we recruit 150 Mechanical Turk workers with a payout of $1 per HIT.

Testing and Performance Metrics

We test agent $\mathcal{A}_{3}$ with parser $\mathcal{P}_{3}$ and perception models $P_{c,3}$ against unseen tasks,888Empirically, parser $\mathcal{P}_{4}$ overfits the training data, so we evaluate with $\mathcal{P}_{3}$. For $\mathcal{A}_{4}^{*}$, this is not a concern since the initial parser parameters $\mathcal{P}_{1}^{*}$ are used. and denote it Trained (Parsing+Perception). We also test an ablation agent, $\mathcal{A}_{4}^{*}$, with parser $\mathcal{P}_{1}^{*}$ and perception models $P_{c,4}$ (trained perception models with an initial, baseline parser with parsing rules only added for new concept model words), and denote it Trained (Perception). These agents are compared against the baseline agent $\mathcal{A}_{1}$, denoted Initial (Seed).

We measure the number of clarification questions asked during the dialog to accomplish the task correctly. This metric should decrease as the agent refines its parsing and perception modules, needing to ask fewer questions about the unseen locations and objects in the test tasks. We also consider users’ answers to survey questions about usability. Each question was answered on a 7-point Likert scale: from Strongly Disagree (1) to Strongly Agree (7).

IV-B Mechanical Turk Evaluation

We prompt users with instructions like: Give the robot a command to solve this problem: The robot should be at the X marked on the green map, with a green-highlighted map marking the target. Users are instructed to command the robot to perform a navigation, delivery, and relocation task in that order. The simple, simulated environment in which the instructions were grounded reflects a physical office space, allowing us to transfer the learned agent into an embodied robot (Section IV-C). Users type answers to agent questions or select them from menus (Figure 3). For delivery and relocation, target objects are given as pictures. Pictures are also shown alongside concept questions like Would you use the word ‘rattling’ when describing this object?

Table II gives measures of the agents’ performance in terms of the number of clarification questions asked before reaching the correct task specification to perform. For both navigation and relocation, there is a slight decrease in the number of questions between the Initial agent and the Trained (Parsing+Perception) agent. The Trained (Perception) agent which only retrains and adds new concept models from conversation history sees slightly worse performance across tasks, possibly due to a larger lexicon of adjectives and nouns (e.g., can as a descriptive noun now polysemous with can as a verb—can you…) without corresponding parsing updates. None of these differences are statistically significant, possibly because comparatively few users completed tasks correctly, necessary to use this metric.999Across agents, an average of 42%, 39%, and 9.5% workers completed navigation, delivery, and relocation tasks correctly, respectively. A necessary step in future studies is to improve worker success rates, possibly through easier interfaces, faster HITs, and higher payouts.

Table III gives measures of the agents’ performance in terms of qualitative survey prompt responses from workers. Prompts were: I would use a robot like this to help navigate a new building, I would use a robot like this to get items for myself or others, and I would use a robot like this to move items from place to place. Across tasks, the Trained (Parsing+Perception) agent novel to this work is rated as more usable than both the Initial agent and the Trained (Perception) agent that updated only its concept models from training conversations.

The agent acquired new perceptual concept models (25 in total), and synonym words for existing concepts during training. Figure 4 shows the learned model for can on unseen test objects. The agent’s ordering of test objects’ can-ness qualitatively matches intuition.

IV-C Physical Robot Implementation

The browser-interfaced, Mechanical Turk agent enabled us to collect controlled training data, but our end goal is a human-robot interface in a physically shared environment. To establish that the agent and learning pipeline are robust and efficient enough to operate on real hardware in a live setting, we complement our Mechanical Turk evaluation with a demonstration of an embodied robot agent (Figure 5).

We use the BWIBot [39, 40], which can perceive and manipulate objects (Xtion ASUS Pro camera, Kinova MICO arm), navigate autonomously (Hokuyo lidar), record and transcribe human speech (Blue Snowball microphone, Google Speech API101010https://cloud.google.com/speech/), and verbalize audio responses (Festival Speech Synthesis System111111http://www.cstr.ed.ac.uk/projects/festival/). Tabletop perception is implemented with RANSAC [41] plane fitting and Euclidean clustering as provided by the Point Cloud Library [42].

The agent is trained on Mechanical Turk conversations, transferring learned linguistic (e.g., lounge by the conference room) and perceptual (e.g., object classes like can) knowledge across platforms from simple simulation to real world application. In a live human-robot dialog, an experimenter tells the agent to move a rattling container from the lounge by the conference room to bob’s office, requiring the agent to select correct rooms and to learn the new, audio-grounded word rattling from the human user.121212Demonstration video: https://youtu.be/PbOfteZ_CJc?t=5.

V Conclusion

This paper proposes a robotic agent that leverages conversations with humans to expand small, hand-crafted language understanding resources both for translating natural language commands to abstract semantic forms and for grounding those abstractions to physical object properties. We make several key assumptions, and promising areas of future work involve removing or weakening those assumptions. In this work, the actions the robot can perform can be broken down into tuples of discrete semantic roles (e.g., patient, source), but, in general, robot agents need to reason about more continuous action spaces, and to acquire new, previously unseen actions from conversations with humans [15]. When learning from conversations, we also assume the human user is cooperative and truthful, but detecting and dealing with combative users is necessary for real world deployment, and would improve learning quality from Mechanical Turk dialogs. Making a closed world assumption, our agent has explored all available objects in the environment, but detecting and exploring objects on the fly using only task relevant behaviors [43, 44] would remove this restriction. Finally, dealing with complex adjective-noun dependencies (e.g., a fake gun is fake but is not a gun) and graded adjectives (e.g., a heavy mug weighs less than a light suitcase) is necessary to move beyond simple, categorical object properties like can.

We hope that our agent and learning strategies for an end-to-end dialog system with perceptual connections to the real world inspire further research on grounded human-robot dialog for command understanding.

Acknowledgements

This work was supported by a National Science Foundation Graduate Research Fellowship to the first author, an NSF EAGER grant (IIS-1548567), and an NSF NRI grant (IIS-1637736). This work has taken place in the Learning Agents Research Group (LARG) at UT Austin. LARG research is supported in part by NSF (CNS-1305287, IIS-1637736, IIS-1651089, IIS-1724157), TxDOT, Intel, Raytheon, and Lockheed Martin.