Multimodal Continuation-style Architectures for Human-Robot Interaction

TL;DR

This paper introduces a multimodal, continuation-style architecture for real-time human-robot interaction that effectively integrates speech and gesture inputs to understand and respond in virtual environments.

Contribution

It presents a novel architecture combining pushdown automata and machine learning outputs for multimodal, real-time interaction in virtual worlds.

Findings

Supports incremental understanding of multimodal input

Enables context-aware question asking

Facilitates multimodal one-shot learning of gestures

Abstract

We present an architecture for integrating real-time, multimodal input into a computational agent's contextual model. Using a human-avatar interaction in a virtual world, we treat aligned gesture and speech as an ensemble where content may be communicated by either modality. With a modified nondeterministic pushdown automaton architecture, the computer system: (1) consumes input incrementally using continuation-passing style until it achieves sufficient understanding the user's aim; (2) constructs and asks questions where necessary using established contextual information; and (3) maintains track of prior discourse items using multimodal cues. This type of architecture supports special cases of pushdown and finite state automata as well as integrating outputs from machine learning models. We present examples of this architecture's use in multimodal one-shot learning interactions of novel gestures and live action composition.