Encoding Emotion Through Self-Supervised Eye Movement Reconstruction

TL;DR

This paper introduces a self-supervised eye movement reconstruction model that predicts emotional expressions from naturalistic, low-resolution videos, demonstrating its effectiveness in encoding affective signals and predicting emotional behaviors.

Contribution

It presents a novel self-supervised gaze detection model inspired by language pretraining, enabling emotion prediction from unlabeled video data.

Findings

Model predicts emotional states from eye movements.

Pretraining performance correlates with emotion prediction accuracy.

Effective use of unlabeled videos for emotion-related gaze analysis.

Abstract

The relationship between emotional expression and eye movement is well-documented, with literature establishing gaze patterns are reliable indicators of emotion. However, most studies utilize specialized, high-resolution eye-tracking equipment, limiting the potential reach of findings. We investigate how eye movement can be used to predict multimodal markers of emotional expression from naturalistic, low-resolution videos. We utilize a collection of video interviews from the USC Shoah Foundation's Visual History Archive with Holocaust survivors as they recount their experiences in the Auschwitz concentration camp. Inspired by pretraining methods on language models, we develop a novel gaze detection model that uses self-supervised eye movement reconstruction that can effectively leverage unlabeled video. We use this model's encoder embeddings to fine-tune models on two downstream tasks related to emotional expression. The first is aligning eye movement with directional emotion estimates from speech. The second task is using eye gaze as a predictor of three momentary manifestations of emotional behaviors: laughing, crying/sobbing, and sighing. We find our new model is predictive of emotion outcomes and observe a positive correlation between pretraining performance and emotion processing performance for both experiments. We conclude self-supervised eye movement reconstruction is an effective method for encoding the affective signal they carry.