A Model-Agnostic Approach for Semantically Driven Disambiguation in Human-Robot Interaction

TL;DR

This paper introduces a model-agnostic, semantically driven clarification approach that improves a robot's ability to disambiguate user instructions and locate objects more efficiently in shared environments.

Contribution

It presents a novel, flexible clarification method that leverages various knowledge embedding models and enhances object search accuracy in ambiguous human-robot interactions.

Findings

Clarifications significantly improve object localization success rates.

The approach is compatible with different semantic encoders and large language models.

Robots can locate objects on the first attempt more frequently.

Abstract

Ambiguities are inevitable in human-robot interaction, especially when a robot follows user instructions in a large, shared space. For example, if a user asks the robot to find an object in a home environment with underspecified instructions, the object could be in multiple locations depending on missing factors. For instance, a bowl might be in the kitchen cabinet or on the dining room table, depending on whether it is clean or dirty, full or empty, and the presence of other objects around it. Previous works on object search have assumed that the queried object is immediately visible to the robot or have predicted object locations using one-shot inferences, which are likely to fail for ambiguous or partially understood instructions. This paper focuses on these gaps and presents a novel model-agnostic approach leveraging semantically driven clarifications to enhance the robot's ability to locate queried objects in fewer attempts. Specifically, we leverage different knowledge embedding models, and when ambiguities arise, we propose an informative clarification method, which follows an iterative prediction process. The user experiment evaluation of our method shows that our approach is applicable to different custom semantic encoders as well as LLMs, and informative clarifications improve performances, enabling the robot to locate objects on its first attempts. The user experiment data is publicly available at https://github.com/IrmakDogan/ExpressionDataset.