Exploring a Multimodal Fusion-based Deep Learning Network for Detecting Facial Palsy

TL;DR

This study develops a multimodal deep learning model combining image and structured facial data to improve the detection of facial palsy, demonstrating the benefits of fusion approaches in clinical assessment.

Contribution

The paper introduces a novel multimodal fusion-based deep learning framework that integrates unstructured and structured data for facial palsy detection, with experimental validation on patient videos.

Findings

Facial expression features achieved the highest precision (76.22%).

Facial line segment images achieved the highest recall (83.47%).

Multimodal fusion slightly improved precision to 77.05%.

Abstract

Algorithmic detection of facial palsy offers the potential to improve current practices, which usually involve labor-intensive and subjective assessment by clinicians. In this paper, we present a multimodal fusion-based deep learning model that utilizes unstructured data (i.e. an image frame with facial line segments) and structured data (i.e. features of facial expressions) to detect facial palsy. We then contribute to a study to analyze the effect of different data modalities and the benefits of a multimodal fusion-based approach using videos of 21 facial palsy patients. Our experimental results show that among various data modalities (i.e. unstructured data - RGB images and images of facial line segments and structured data - coordinates of facial landmarks and features of facial expressions), the feed-forward neural network using features of facial expression achieved the highest precision of 76.22 while the ResNet-based model using images of facial line segments achieved the highest recall of 83.47. When we leveraged both images of facial line segments and features of facial expressions, our multimodal fusion-based deep learning model slightly improved the precision score to 77.05 at the expense of a decrease in the recall score.