Switching Head-Tail Funnel UNITER for Dual Referring Expression Comprehension with Fetch-and-Carry Tasks

TL;DR

This paper introduces Switching Head-Tail Funnel UNITER, a model that efficiently understands natural language instructions for domestic robots by predicting objects and destinations separately, improving accuracy and success rates in real-world tasks.

Contribution

The paper presents a novel model that reduces computational complexity by predicting target objects and destinations separately, validated on a new dataset and real robot experiments.

Findings

Outperforms baseline in language comprehension accuracy

Achieves over 90% success rate in object delivery tasks

Effective in real-world domestic robot scenarios

Abstract

This paper describes a domestic service robot (DSR) that fetches everyday objects and carries them to specified destinations according to free-form natural language instructions. Given an instruction such as "Move the bottle on the left side of the plate to the empty chair," the DSR is expected to identify the bottle and the chair from multiple candidates in the environment and carry the target object to the destination. Most of the existing multimodal language understanding methods are impractical in terms of computational complexity because they require inferences for all combinations of target object candidates and destination candidates. We propose Switching Head-Tail Funnel UNITER, which solves the task by predicting the target object and the destination individually using a single model. Our method is validated on a newly-built dataset consisting of object manipulation instructions and semi photo-realistic images captured in a standard Embodied AI simulator. The results show that our method outperforms the baseline method in terms of language comprehension accuracy. Furthermore, we conduct physical experiments in which a DSR delivers standardized everyday objects in a standardized domestic environment as requested by instructions with referring expressions. The experimental results show that the object grasping and placing actions are achieved with success rates of more than 90%.