# Human-Centered Emotion Recognition in Animated GIFs

## TL;DR

This paper introduces a human-centered approach to GIF emotion recognition using a novel neural network that emphasizes facial features and temporal dynamics, achieving superior performance and interpretability.

## Contribution

The study proposes the Keypoint Attended Visual Attention Network (KAVAN), integrating facial attention and hierarchical temporal modeling for improved GIF emotion recognition.

## Key findings

- Outperforms state-of-the-art on MIT GIFGIF dataset
- Facial attention enhances recognition accuracy and interpretability
- Hierarchical segment modeling captures global GIF features

## Abstract

As an intuitive way of expression emotion, the animated Graphical Interchange Format (GIF) images have been widely used on social media. Most previous studies on automated GIF emotion recognition fail to effectively utilize GIF's unique properties, and this potentially limits the recognition performance. In this study, we demonstrate the importance of human related information in GIFs and conduct human-centered GIF emotion recognition with a proposed Keypoint Attended Visual Attention Network (KAVAN). The framework consists of a facial attention module and a hierarchical segment temporal module. The facial attention module exploits the strong relationship between GIF contents and human characters, and extracts frame-level visual feature with a focus on human faces. The Hierarchical Segment LSTM (HS-LSTM) module is then proposed to better learn global GIF representations. Our proposed framework outperforms the state-of-the-art on the MIT GIFGIF dataset. Furthermore, the facial attention module provides reliable facial region mask predictions, which improves the model's interpretability.