Cross-modality Force and Language Embeddings for Natural Human-Robot Communication

TL;DR

This paper introduces a novel framework for embedding verbal and force profile data into a shared space to enhance natural human-robot communication through coordinated verbal and haptic cues.

Contribution

It presents a new method for cross-modality embedding of language and force profiles, enabling effective integration and substitution in human-robot interaction.

Findings

Language and force profiles can be embedded in a unified latent space.

Proximity in the latent space indicates effective coordination.

The framework supports supplementing, integrating, and substituting communication modalities.

Abstract

A method for cross-modality embedding of force profile and words is presented for synergistic coordination of verbal and haptic communication. When two people carry a large, heavy object together, they coordinate through verbal communication about the intended movements and physical forces applied to the object. This natural integration of verbal and physical cues enables effective coordination. Similarly, human-robot interaction could achieve this level of coordination by integrating verbal and haptic communication modalities. This paper presents a framework for embedding words and force profiles in a unified manner, so that the two communication modalities can be integrated and coordinated in a way that is effective and synergistic. Here, it will be shown that, although language and physical force profiles are deemed completely different, the two can be embedded in a unified latent space and proximity between the two can be quantified. In this latent space, a force profile and words can a) supplement each other, b) integrate the individual effects, and c) substitute in an exchangeable manner. First, the need for cross-modality embedding is addressed, and the basic architecture and key building block technologies are presented. Methods for data collection and implementation challenges will be addressed, followed by experimental results and discussions.