Deep Learning-Based Real-Time Sequential Facial Expression Analysis Using Geometric Features

TL;DR

This paper introduces a fast, accurate deep learning method for real-time sequential facial expression recognition using geometric features and landmarks, suitable for human-computer interaction and emotion-aware systems.

Contribution

It presents a novel approach combining MediaPipe FaceMesh, geometric features, and ConvLSTM1D for real-time facial expression analysis, demonstrating high accuracy and speed.

Findings

Achieved up to 93% accuracy on CK+ dataset

Processed approximately 165 frames per second

Demonstrated good generalization across multiple datasets

Abstract

Facial expression recognition is a crucial component in enhancing human-computer interaction and developing emotion-aware systems. Real-time detection and interpretation of facial expressions have become increasingly important for various applications, from user experience personalization to intelligent surveillance systems. This study presents a novel approach to real-time sequential facial expression recognition using deep learning and geometric features. The proposed method utilizes MediaPipe FaceMesh for rapid and accurate facial landmark detection. Geometric features, including Euclidean distances and angles, are extracted from these landmarks. Temporal dynamics are incorporated by analyzing feature differences between consecutive frames, enabling the detection of onset, apex, and offset phases of expressions. For classification, a ConvLSTM1D network followed by multilayer perceptron blocks is employed. The method's performance was evaluated on multiple publicly available datasets, including CK+, Oulu-CASIA (VIS and NIR), and MMI. Accuracies of 93%, 79%, 77%, and 68% were achieved respectively. Experiments with composite datasets were also conducted to assess the model's generalization capabilities. The approach demonstrated real-time applicability, processing approximately 165 frames per second on consumer-grade hardware. This research contributes to the field of facial expression analysis by providing a fast, accurate, and adaptable solution. The findings highlight the potential for further advancements in emotion-aware technologies and personalized user experiences, paving the way for more sophisticated human-computer interaction systems. To facilitate further research in this field, the complete source code for this study has been made publicly available on GitHub: https://github.com/miralab-ai/facial-expression-analysis.