Generating and Estimating Nonverbal Alphabets for Situated and Multimodal Communications

TL;DR

This paper presents a formalized approach for generating and estimating nonverbal symbols and alphabets using multimodal data and machine learning, enhancing situated and multimodal communication for diverse applications.

Contribution

It introduces a framework and methods for creating and evaluating nonverbal alphabets tailored to user needs, integrating multimodal sensors and machine learning techniques.

Findings

Multimodal data improves understanding of nonverbal communication effectiveness.

Generated symbols can be applied in human-AI interaction and assistive technologies.

Machine learning enhances estimation of symbol usefulness in real-world scenarios.

Abstract

In this paper, we discuss the formalized approach for generating and estimating symbols (and alphabets), which can be communicated by the wide range of non-verbal means based on specific user requirements (medium, priorities, type of information that needs to be conveyed). The short characterization of basic terms and parameters of such symbols (and alphabets) with approaches to generate them are given. Then the framework, experimental setup, and some machine learning methods to estimate usefulness and effectiveness of the nonverbal alphabets and systems are presented. The previous results demonstrate that usage of multimodal data sources (like wearable accelerometer, heart monitor, muscle movements sensors, braincomputer interface) along with machine learning approaches can provide the deeper understanding of the usefulness and effectiveness of such alphabets and systems for nonverbal and situated communication. The symbols (and alphabets) generated and estimated by such methods may be useful in various applications: from synthetic languages and constructed scripts to multimodal nonverbal and situated interaction between people and artificial intelligence systems through Human-Computer Interfaces, such as mouse gestures, touchpads, body gestures, eyetracking cameras, wearables, and brain-computing interfaces, especially in applications for elderly care and people with disabilities.