Detecting Cognitive Appraisals from Facial Expressions for Interest Recognition

TL;DR

This study explores the facial expressions linked to interest and related appraisals, demonstrating the feasibility of detecting cognitive appraisals from facial cues using machine learning, despite challenges in expression variability.

Contribution

First to show that cognitive appraisals like interest and curiosity can be detected from facial expressions using automated analysis and machine learning models.

Findings

Facial expressions vary significantly between individuals in response to interesting content.

Automated facial feature analysis can predict levels of interest and related appraisals.

Detecting interest from facial expressions remains challenging due to subtle and inconsistent reactions.

Abstract

Interest makes one hold her attention on the object of interest. Automatic recognition of interest has numerous applications in human-computer interaction. In this paper, we study the facial expressions associated with interest and its underlying and closely related components, namely, curiosity, coping potential, novelty and complexity. To this end, we conducted an experiment in which participants watched images and micro-videos while a front-facing camera recorded their expressions. After watching each item they self-reported their level of interest, curiosity, coping potential and perceived novelty and complexity. Using an automated method, we tracked facial action units (AU) and studied the relationship between the presence of facial movements with interest and its related components. We then tracked the facial landmarks, e.g., corners of lips, and extracted features from each response. We trained random forests regression models to detect the level of interest, curiosity, and appraisals. We found a large difference between the way people report and react to interesting visual content. The expressions in response to images and micro-videos were not always pronounced depending on the participants. This makes the direct detection of interest from facial expressions a challenging problem. With this work, for the first time, we demonstrate the feasibility of detecting cognitive appraisals from facial expressions which will open the door for appraisal-driven emotion recognition methods.