Spatial and Visual Perspective-Taking via View Rotation and Relation Reasoning for Embodied Reference Understanding

TL;DR

This paper introduces REP, a novel method for embodied reference understanding that models spatial and visual perspective-taking through view rotation and relation reasoning, significantly improving accuracy in locating objects based on language and gesture.

Contribution

The paper proposes a new approach combining view rotation and relation reasoning to better model perspective-taking in embodied reference understanding.

Findings

REP outperforms existing methods by +5.22% accuracy on YouRefIt.

View rotation effectively aligns egocentric views with sender perspectives.

Relation reasoning enhances multi-modal understanding of sender-object relations.

Abstract

Embodied Reference Understanding studies the reference understanding in an embodied fashion, where a receiver is required to locate a target object referred to by both language and gesture of the sender in a shared physical environment. Its main challenge lies in how to make the receiver with the egocentric view access spatial and visual information relative to the sender to judge how objects are oriented around and seen from the sender, i.e., spatial and visual perspective-taking. In this paper, we propose a REasoning from your Perspective (REP) method to tackle the challenge by modeling relations between the receiver and the sender and the sender and the objects via the proposed novel view rotation and relation reasoning. Specifically, view rotation first rotates the receiver to the position of the sender by constructing an embodied 3D coordinate system with the position of the sender as the origin. Then, it changes the orientation of the receiver to the orientation of the sender by encoding the body orientation and gesture of the sender. Relation reasoning models the nonverbal and verbal relations between the sender and the objects by multi-modal cooperative reasoning in gesture, language, visual content, and spatial position. Experiment results demonstrate the effectiveness of REP, which consistently surpasses all existing state-of-the-art algorithms by a large margin, i.e., +5.22% absolute accuracy in terms of Prec0.5 on YouRefIt.