Multimodal Characterization of Emotion within Multimedia Space

TL;DR

This paper explores multimodal emotion detection in multimedia content, demonstrating improved accuracy over unimodal methods and analyzing emotional coherence across different modalities using deep learning.

Contribution

It introduces a comprehensive approach to classify emotions across multiple modalities and characterizes multimedia content based on emotional content.

Findings

Multimodal emotion detection outperforms unimodal solutions.

Deep learning models effectively classify emotions across modalities.

Analysis of emotional coherence enhances understanding of multimedia content.

Abstract

Technological advancement and its omnipresent connection have pushed humans past the boundaries and limitations of a computer screen, physical state, or geographical location. It has provided a depth of avenues that facilitate human-computer interaction that was once inconceivable such as audio and body language detection. Given the complex modularities of emotions, it becomes vital to study human-computer interaction, as it is the commencement of a thorough understanding of the emotional state of users and, in the context of social networks, the producers of multimodal information. This study first acknowledges the accuracy of classification found within multimodal emotion detection systems compared to unimodal solutions. Second, it explores the characterization of multimedia content produced based on their emotions and the coherence of emotion in different modalities by utilizing deep learning models to classify emotion across different modalities.