Toward using Speech to Sense Student Emotion in Remote Learning Environments

TL;DR

This paper investigates the feasibility of using speech from self-control tasks to automatically sense student emotions in remote learning, aiming to enhance learning experiences through emotion-aware technology.

Contribution

It introduces a new dataset and demonstrates that speech variations in self-control tasks can predict student emotions, enabling emotion sensing in remote learning environments.

Findings

Speech-based self-control tasks show perceptible emotion variation.

Automatic prediction of valence, arousal, and dominance is feasible.

Speech cues can be integrated into remote learning for improved feedback.

Abstract

With advancements in multimodal communication technologies, remote learning environments such as, distance universities are increasing. Remote learning typically happens asynchronously. As a consequence, unlike face-to-face in-person classroom teaching, this lacks availability of sufficient emotional cues for making learning a pleasant experience. Motivated by advances made in the paralinguistic speech processing community on emotion prediction, in this paper we explore use of speech for sensing students' emotions by building upon speech-based self-control tasks developed to aid effective remote learning. More precisely, we investigate: (a) whether speech acquired through self-control tasks exhibit perceptible variation along valence, arousal, and dominance dimensions? and (b) whether those dimensional emotion variations can be automatically predicted? We address these two research questions by developing a dataset containing spontaneous monologue speech acquired as open responses to self-control tasks and by carrying out subjective listener evaluations and automatic dimensional emotion prediction studies on that dataset. Our investigations indicate that speech-based self-control tasks can be a means to sense student emotion in remote learning environment. This opens potential venues to seamlessly integrate paralinguistic speech processing technologies in the remote learning loop for enhancing learning experiences through instructional design and feedback generation.