Do Smart Glasses Dream of Sentimental Visions? Deep Emotionship Analysis for Eyewear Devices

TL;DR

This paper introduces EMOShip, a deep learning system for smart eyewear that not only recognizes user emotions with high accuracy but also semantically analyzes their causes through visual perceptions, enhancing emotion understanding.

Contribution

The paper presents EMOShip, a novel system that jointly performs emotion recognition and semantic analysis of visual perceptions, addressing limitations of prior emotion recognition methods.

Findings

Achieves 80.2% emotion recognition accuracy, outperforming existing methods.

Demonstrates the system's ability to analyze emotional causes through visual perception.

Potential applications include emotion self-reflection and life-logging.

Abstract

Emotion recognition in smart eyewear devices is highly valuable but challenging. One key limitation of previous works is that the expression-related information like facial or eye images is considered as the only emotional evidence. However, emotional status is not isolated; it is tightly associated with people's visual perceptions, especially those sentimental ones. However, little work has examined such associations to better illustrate the cause of different emotions. In this paper, we study the emotionship analysis problem in eyewear systems, an ambitious task that requires not only classifying the user's emotions but also semantically understanding the potential cause of such emotions. To this end, we devise EMOShip, a deep-learning-based eyewear system that can automatically detect the wearer's emotional status and simultaneously analyze its associations with semantic-level visual perceptions. Experimental studies with 20 participants demonstrate that, thanks to the emotionship awareness, EMOShip not only achieves superior emotion recognition accuracy over existing methods (80.2% vs. 69.4%), but also provides a valuable understanding of the cause of emotions. Pilot studies with 20 participants further motivate the potential use of EMOShip to empower emotion-aware applications, such as emotionship self-reflection and emotionship life-logging.