PROGrasp: Pragmatic Human-Robot Communication for Object Grasping

TL;DR

PROGrasp introduces a new pragmatic human-robot interaction framework for object grasping, enabling robots to interpret intentions from context and dialogue, improving target identification in interactive scenarios.

Contribution

The paper presents PROGrasp, a novel system that incorporates pragmatic inference for human-robot communication in object grasping tasks, along with a new dataset and task scenario.

Findings

Effective in offline target discovery

Successful online interaction with robot arm

Improved understanding of pragmatic cues

Abstract

Interactive Object Grasping (IOG) is the task of identifying and grasping the desired object via human-robot natural language interaction. Current IOG systems assume that a human user initially specifies the target object's category (e.g., bottle). Inspired by pragmatics, where humans often convey their intentions by relying on context to achieve goals, we introduce a new IOG task, Pragmatic-IOG, and the corresponding dataset, Intention-oriented Multi-modal Dialogue (IM-Dial). In our proposed task scenario, an intention-oriented utterance (e.g., "I am thirsty") is initially given to the robot. The robot should then identify the target object by interacting with a human user. Based on the task setup, we propose a new robotic system that can interpret the user's intention and pick up the target object, Pragmatic Object Grasping (PROGrasp). PROGrasp performs Pragmatic-IOG by incorporating modules for visual grounding, question asking, object grasping, and most importantly, answer interpretation for pragmatic inference. Experimental results show that PROGrasp is effective in offline (i.e., target object discovery) and online (i.e., IOG with a physical robot arm) settings. Code and data are available at https://github.com/gicheonkang/prograsp.