Heart rates, facial expressions and self-reports: a multimodal longitudinal approach of learners' emotions in the foreign language classroom

TL;DR

This study uses heart rate, facial expressions, and self-reports to track learners' emotions in foreign language classrooms over time, revealing how emotions change during lessons.

Contribution

The study introduces a multimodal longitudinal method to capture individual emotional patterns in real classroom settings.

Findings

Positive emotional contagion was observed during peer interactions in language learning.

Multimodal data revealed transient emotional states linked to classroom activities and interactions.

Individualized emotional patterns highlight the variability in learners' emotional experiences.

Abstract

Emotions in educational settings are often studied through self-reports or lab experiments, limiting insights into their real-world dynamics. This study examines learner emotions in authentic foreign language classrooms using a multimodal longitudinal approach. Over 16 consecutive sessions, we collected heart rate (HR) signals, emotional facial expressions (EFE), classroom observations, and self-reports on enjoyment, anxiety, and boredom to capture both physiological and self-perceived emotional responses. Rather than aggregating data across students, we focused on individualized emotional patterns to understand variations in emotional experiences. Each dataset included extensive video recordings, continuous HR monitoring, detailed observational notes, and post-session questionnaires, providing a high-resolution picture of emotional dynamics. Using unsupervised clustering techniques, we identified key emotional episodes—peaks and drops in physiological arousal (heart rate variation) and facial expression—relative to individual emotional baselines. These moments were cross-referenced with classroom observations and self-reports for validation. Findings highlight moments of positive emotional contagion during peer interactions, emphasizing the social dimension of language learning. This multimodal approach captures the interplay of physiological, behavioral, and subjective responses, offering a scalable method for studying classroom emotions. Methodologically, it demonstrates how multimodal analytics can uncover transient emotional states in real-world settings, while practically informing adaptive teaching strategies, such as leveraging peer interactions to enhance engagement or reduce anxiety. By integrating physiological, behavioral, and subjective data, this study provides a comprehensive framework for understanding the affective dimensions of learning.

Figures