DoRO: Disambiguation of referred object for embodied agents

TL;DR

This paper introduces DoRO, a system enabling mobile robots to disambiguate referred objects by aggregating multi-view observations and asking targeted queries, improving accuracy in object grounding tasks.

Contribution

The novel DoRO system allows embodied agents to disambiguate objects through multi-view aggregation and adaptive querying, addressing limitations of static view-based methods.

Findings

Improves ambiguity detection accuracy in simulated environments.

Generates more informative and precise queries for disambiguation.

Enhances object grounding performance in mobile robotic scenarios.

Abstract

Robotic task instructions often involve a referred object that the robot must locate (ground) within the environment. While task intent understanding is an essential part of natural language understanding, less effort is made to resolve ambiguity that may arise while grounding the task. Existing works use vision-based task grounding and ambiguity detection, suitable for a fixed view and a static robot. However, the problem magnifies for a mobile robot, where the ideal view is not known beforehand. Moreover, a single view may not be sufficient to locate all the object instances in the given area, which leads to inaccurate ambiguity detection. Human intervention is helpful only if the robot can convey the kind of ambiguity it is facing. In this article, we present DoRO (Disambiguation of Referred Object), a system that can help an embodied agent to disambiguate the referred object by raising a suitable query whenever required. Given an area where the intended object is, DoRO finds all the instances of the object by aggregating observations from multiple views while exploring & scanning the area. It then raises a suitable query using the information from the grounded object instances. Experiments conducted with the AI2Thor simulator show that DoRO not only detects the ambiguity more accurately but also raises verbose queries with more accurate information from the visual-language grounding.