REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?

TL;DR

This paper introduces REI-Bench, a benchmark for robot task planning with vague human instructions, revealing significant performance drops due to referential vagueness and proposing context cognition to improve understanding.

Contribution

It is the first benchmark modeling vague referring expressions grounded in pragmatic theory and proposes a context cognition method to enhance robot task planning accuracy.

Findings

Vagueness in referring expressions can reduce planning success by up to 36.9%.

Most planning failures are due to missing objects in the robot's understanding.

Context cognition significantly improves robot understanding of vague instructions.

Abstract

Robot task planning decomposes human instructions into executable action sequences that enable robots to complete a series of complex tasks. Although recent large language model (LLM)-based task planners achieve amazing performance, they assume that human instructions are clear and straightforward. However, real-world users are not experts, and their instructions to robots often contain significant vagueness. Linguists suggest that such vagueness frequently arises from referring expressions (REs), whose meanings depend heavily on dialogue context and environment. This vagueness is even more prevalent among the elderly and children, who are the groups that robots should serve more. This paper studies how such vagueness in REs within human instructions affects LLM-based robot task planning and how to overcome this issue. To this end, we propose the first robot task planning benchmark that systematically models vague REs grounded in pragmatic theory (REI-Bench), where we discover that the vagueness of REs can severely degrade robot planning performance, leading to success rate drops of up to 36.9%. We also observe that most failure cases stem from missing objects in planners. To mitigate the REs issue, we propose a simple yet effective approach: task-oriented context cognition, which generates clear instructions for robots, achieving state-of-the-art performance compared to aware prompts, chains of thought, and in-context learning. By tackling the overlooked issue of vagueness, this work contributes to the research community by advancing real-world task planning and making robots more accessible to non-expert users, e.g., the elderly and children.