Emotion recognition in talking-face videos using persistent entropy and neural networks

TL;DR

This paper introduces a novel method combining persistent entropy and neural networks to classify emotions from talking-face videos by analyzing topology signatures derived from audio and visual data.

Contribution

It proposes a new approach that uses persistent entropy to generate topology signatures from videos, improving emotion recognition accuracy over existing methods.

Findings

Achieved competitive accuracy in emotion classification.

Demonstrated robustness of topology signatures to small video changes.

Outperformed several state-of-the-art methods.

Abstract

The automatic recognition of a person's emotional state has become a very active research field that involves scientists specialized in different areas such as artificial intelligence, computer vision or psychology, among others. Our main objective in this work is to develop a novel approach, using persistent entropy and neural networks as main tools, to recognise and classify emotions from talking-face videos. Specifically, we combine audio-signal and image-sequence information to compute a topology signature(a 9-dimensional vector) for each video. We prove that small changes in the video produce small changes in the signature. These topological signatures are used to feed a neural network to distinguish between the following emotions: neutral, calm, happy, sad, angry, fearful, disgust, and surprised. The results reached are promising and competitive, beating the performance reached in other state-of-the-art works found in the literature.