Realtime Multimodal Emotion Estimation using Behavioral and Neurophysiological Data

TL;DR

This paper introduces a real-time multimodal emotion estimation system that combines neurophysiological and behavioral data to support inclusive emotion recognition, especially for neurodivergent individuals, with applications in education and interaction support.

Contribution

It presents a novel integrated system that fuses neurophysiological and behavioral modalities for real-time, interpretable emotion estimation tailored for neurodiverse users.

Findings

Demonstrated system effectiveness in passive media viewing scenarios.

Captured real-time facial and vocal expressions during semi-scripted conversations.

Enabled personalized emotion feedback for neurodiversity support.

Abstract

Many individuals especially those with autism spectrum disorder (ASD), alexithymia, or other neurodivergent profiles face challenges in recognizing, expressing, or interpreting emotions. To support more inclusive and personalized emotion technologies, we present a real-time multimodal emotion estimation system that combines neurophysiological EEG, ECG, blood volume pulse (BVP), and galvanic skin response (GSR/EDA) and behavioral modalities (facial expressions, and speech) in a unified arousal-valence 2D interface to track moment-to-moment emotional states. This architecture enables interpretable, user-specific analysis and supports applications in emotion education, neuroadaptive feedback, and interaction support for neurodiverse users. Two demonstration scenarios illustrate its application: (1) passive media viewing (2D or VR videos) reveals cortical and autonomic responses to affective content, and (2) semi-scripted conversations with a facilitator or virtual agent capture real-time facial and vocal expressions. These tasks enable controlled and naturalistic emotion monitoring, making the system well-suited for personalized feedback and neurodiversity-informed interaction design.