On the Linguistic and Computational Requirements for Creating Face-to-Face Multimodal Human-Machine Interaction

TL;DR

This paper analyzes human-avatar face-to-face interactions to identify linguistic and structural requirements for multimodal interfaces, emphasizing the need for higher-level control layers informed by conversation analysis and cognitive science.

Contribution

It introduces a framework incorporating Conversation Analysis and Theory of Mind to enhance multimodal human-machine interaction interfaces.

Findings

Double-loop feedback occurs in face-to-face conversations.

Multimodal actions are distributed between speaker and listener.

A new control layer is needed for social context and interaction planning.

Abstract

In this study, conversations between humans and avatars are linguistically, organizationally, and structurally analyzed, focusing on what is necessary for creating face-to-face multimodal interfaces for machines. We videorecorded thirty-four human-avatar interactions, performed complete linguistic microanalysis on video excerpts, and marked all the occurrences of multimodal actions and events. Statistical inferences were applied to data, allowing us to comprehend not only how often multimodal actions occur but also how multimodal events are distributed between the speaker (emitter) and the listener (recipient). We also observed the distribution of multimodal occurrences for each modality. The data show evidence that double-loop feedback is established during a face-to-face conversation. This led us to propose that knowledge from Conversation Analysis (CA), cognitive science, and Theory of Mind (ToM), among others, should be incorporated into the ones used for describing human-machine multimodal interactions. Face-to-face interfaces require an additional control layer to the multimodal fusion layer. This layer has to organize the flow of conversation, integrate the social context into the interaction, as well as make plans concerning 'what' and 'how' to progress on the interaction. This higher level is best understood if we incorporate insights from CA and ToM into the interface system.