Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene

TL;DR

This paper explores using natural human gestures as a communication interface to improve embodied agent navigation, introducing a VR simulation environment and demonstrating that gesture-based cues enhance navigation performance.

Contribution

The study develops Ges-THOR, a VR environment for gesture-based communication, and shows that natural gestures without predefined semantics can significantly improve navigation tasks.

Findings

Gesture cues improve navigation performance

Natural gestures outperform verbal instructions in experiments

Mutual learning of gestures and navigation enhances agent capabilities

Abstract

Human-robot collaboration is an essential research topic in artificial intelligence (AI), enabling researchers to devise cognitive AI systems and affords an intuitive means for users to interact with the robot. Of note, communication plays a central role. To date, prior studies in embodied agent navigation have only demonstrated that human languages facilitate communication by instructions in natural languages. Nevertheless, a plethora of other forms of communication is left unexplored. In fact, human communication originated in gestures and oftentimes is delivered through multimodal cues, e.g. "go there" with a pointing gesture. To bridge the gap and fill in the missing dimension of communication in embodied agent navigation, we propose investigating the effects of using gestures as the communicative interface instead of verbal cues. Specifically, we develop a VR-based 3D simulation environment, named Ges-THOR, based on AI2-THOR platform. In this virtual environment, a human player is placed in the same virtual scene and shepherds the artificial agent using only gestures. The agent is tasked to solve the navigation problem guided by natural gestures with unknown semantics; we do not use any predefined gestures due to the diversity and versatile nature of human gestures. We argue that learning the semantics of natural gestures is mutually beneficial to learning the navigation task--learn to communicate and communicate to learn. In a series of experiments, we demonstrate that human gesture cues, even without predefined semantics, improve the object-goal navigation for an embodied agent, outperforming various state-of-the-art methods.