# Eye-based Continuous Affect Prediction

## TL;DR

This paper introduces a comprehensive eye-based feature set for continuous affect prediction, demonstrating its effectiveness when combined with speech features and aiming to advance affective computing research.

## Contribution

It proposes a new, refined set of eye-based features from multiple eye channels, enhancing affect prediction accuracy over previous methods.

## Key findings

- Eye-based features improve affect prediction performance.
- Combining eye and speech features yields better results.
- The refined features outperform previous eye-based approaches.

## Abstract

Eye-based information channels include the pupils, gaze, saccades, fixational movements, and numerous forms of eye opening and closure. Pupil size variation indicates cognitive load and emotion, while a person's gaze direction is said to be congruent with the motivation to approach or avoid stimuli. The eyelids are involved in facial expressions that can encode basic emotions. Additionally, eye-based cues can have implications for human annotators of emotions or feelings. Despite these facts, the use of eye-based cues in affective computing is in its infancy, however, and this work is intended to start to address this. Eye-based feature sets, incorporating data from all of the aforementioned information channels, that can be estimated from video are proposed. Feature set refinement is provided by way of continuous arousal and valence learning and prediction experiments on the RECOLA validation set. The eye-based features are then combined with a speech feature set to provide confirmation of their usefulness and assess affect prediction performance compared with group-of-humans-level performance on the RECOLA test set. The core contribution of this paper, a refined eye-based feature set, is shown to provide benefits for affect prediction. It is hoped that this work stimulates further research into eye-based affective computing.