Speejis: Enhancing User Experience of Mobile Voice Messaging with Automatic Visual Speech Emotion Cues

TL;DR

This paper introduces speejis, an automatic visual speech emotion cue system for mobile voice messaging, enhancing user experience by providing accessible emotional expressions through speech analysis.

Contribution

It presents the implementation of speejis and demonstrates its effectiveness in improving user experience in voice messaging through a user study.

Findings

Participants preferred messaging with speejis.

Significant improvements in attractiveness and stimulation.

Finer emotion transition representations achieved.

Abstract

Mobile messaging apps offer an increasing range of emotional expressions, such as emojis to help users manually augment their texting experiences. Accessibility of such augmentations is limited in voice messaging. With the term "speejis" we refer to accessible emojis and other visual speech emotion cues that are created automatically from speech input alone. The paper presents an implementation of speejis and reports on a user study (N=12) comparing the UX of voice messaging with and without speejis. Results show significant differences in measures such as attractiveness and stimulation and a clear preference of all participants for messaging with speejis. We highlight the benefits of using paralinguistic speech processing and continuous emotion models to enable finer grained augmentations of emotion changes and transitions within a single message in addition to augmentations of the overall tone of the message.