Embodied Referring Expression Comprehension in Human-Robot Interaction

TL;DR

This paper introduces the Refer360 dataset and MuRes module to enhance robots' understanding of embodied human instructions through multimodal learning, addressing current limitations in datasets and model performance in diverse HRI settings.

Contribution

The paper presents a large-scale, diverse Refer360 dataset and a novel MuRes module that improves multimodal embodied referring expression comprehension in human-robot interaction.

Findings

MuRes improves model performance across multiple datasets.

Current models inadequately capture embodied interactions.

Refer360 serves as a new benchmark for embodied HRI understanding.

Abstract

As robots enter human workspaces, there is a crucial need for them to comprehend embodied human instructions, enabling intuitive and fluent human-robot interaction (HRI). However, accurate comprehension is challenging due to a lack of large-scale datasets that capture natural embodied interactions in diverse HRI settings. Existing datasets suffer from perspective bias, single-view collection, inadequate coverage of nonverbal gestures, and a predominant focus on indoor environments. To address these issues, we present the Refer360 dataset, a large-scale dataset of embodied verbal and nonverbal interactions collected across diverse viewpoints in both indoor and outdoor settings. Additionally, we introduce MuRes, a multimodal guided residual module designed to improve embodied referring expression comprehension. MuRes acts as an information bottleneck, extracting salient modality-specific signals and reinforcing them into pre-trained representations to form complementary features for downstream tasks. We conduct extensive experiments on four HRI datasets, including the Refer360 dataset, and demonstrate that current multimodal models fail to capture embodied interactions comprehensively; however, augmenting them with MuRes consistently improves performance. These findings establish Refer360 as a valuable benchmark and exhibit the potential of guided residual learning to advance embodied referring expression comprehension in robots operating within human environments.