Situated Multimodal Control of a Mobile Robot: Navigation through a Virtual Environment

TL;DR

This paper introduces a multimodal control interface combining gesture and language for navigating a robot in new environments, utilizing embodied simulation and cross-platform communication.

Contribution

It presents a novel integrated system enabling natural human-robot interaction through coordinated gestures and spoken commands for navigation tasks.

Findings

Effective control of robot navigation via multimodal input

Successful integration of embodied simulation for environment understanding

Enhanced human-robot communication in navigation scenarios

Abstract

We present a new interface for controlling a navigation robot in novel environments using coordinated gesture and language. We use a TurtleBot3 robot with a LIDAR and a camera, an embodied simulation of what the robot has encountered while exploring, and a cross-platform bridge facilitating generic communication. A human partner can deliver instructions to the robot using spoken English and gestures relative to the simulated environment, to guide the robot through navigation tasks.