Color histogram equalization and fine-tuning to improve expression recognition of (partially occluded) faces on sign language datasets

TL;DR

This study enhances facial expression recognition on sign language datasets by applying color normalization and fine-tuning, achieving high accuracy even with partial face data, and comparing performance between upper and lower face regions.

Contribution

Introduces a color normalization and fine-tuning approach to improve expression recognition, especially on partially occluded faces in sign language datasets.

Findings

Achieved 83.8% mean sensitivity in expression recognition.

Lower face recognition (79.6%) outperforms upper face (77.9%).

Upper face recognition exceeds human performance.

Abstract

The goal of this investigation is to quantify to what extent computer vision methods can correctly classify facial expressions on a sign language dataset. We extend our experiments by recognizing expressions using only the upper or lower part of the face, which is needed to further investigate the difference in emotion manifestation between hearing and deaf subjects. To take into account the peculiar color profile of a dataset, our method introduces a color normalization stage based on histogram equalization and fine-tuning. The results show the ability to correctly recognize facial expressions with 83.8% mean sensitivity and very little variance (.042) among classes. Like for humans, recognition of expressions from the lower half of the face (79.6%) is higher than that from the upper half (77.9%). Noticeably, the classification accuracy from the upper half of the face is higher than human level.