Video Multimodal Emotion Recognition System for Real World Applications

TL;DR

This paper presents a multimodal video-based emotion recognition system that integrates multiple AI models to accurately identify speaker emotions at the utterance level, with an interactive interface for user demonstration.

Contribution

It introduces an end-to-end multimodal emotion recognition system that combines various AI models for real-world video applications, enhancing accuracy and user interaction.

Findings

Effective multimodal integration for emotion recognition

End-to-end system achieves accurate utterance-level emotion prediction

Interactive interface demonstrates practical usability

Abstract

This paper proposes a system capable of recognizing a speaker's utterance-level emotion through multimodal cues in a video. The system seamlessly integrates multiple AI models to first extract and pre-process multimodal information from the raw video input. Next, an end-to-end MER model sequentially predicts the speaker's emotions at the utterance level. Additionally, users can interactively demonstrate the system through the implemented interface.