Take an Emotion Walk: Perceiving Emotions from Gaits Using Hierarchical Attention Pooling and Affective Mapping

TL;DR

This paper introduces a semi-supervised autoencoder approach with hierarchical attention pooling and affective mapping to classify emotions from human gait sequences, outperforming existing methods on a benchmark dataset.

Contribution

The novel semi-supervised model effectively captures psychologically-motivated affective features from gait data, improving emotion recognition accuracy over state-of-the-art algorithms.

Findings

Achieved mean average precision of 0.84 on Emotion-Gait dataset.

Outperformed current algorithms by 7-23% in emotion and action recognition.

Significantly improved recognition on less-represented classes by 10-50%.

Abstract

We present an autoencoder-based semi-supervised approach to classify perceived human emotions from walking styles obtained from videos or motion-captured data and represented as sequences of 3D poses. Given the motion on each joint in the pose at each time step extracted from 3D pose sequences, we hierarchically pool these joint motions in a bottom-up manner in the encoder, following the kinematic chains in the human body. We also constrain the latent embeddings of the encoder to contain the space of psychologically-motivated affective features underlying the gaits. We train the decoder to reconstruct the motions per joint per time step in a top-down manner from the latent embeddings. For the annotated data, we also train a classifier to map the latent embeddings to emotion labels. Our semi-supervised approach achieves a mean average precision of 0.84 on the Emotion-Gait benchmark dataset, which contains both labeled and unlabeled gaits collected from multiple sources. We outperform current state-of-art algorithms for both emotion recognition and action recognition from 3D gaits by 7%--23% on the absolute. More importantly, we improve the average precision by 10%--50% on the absolute on classes that each makes up less than 25% of the labeled part of the Emotion-Gait benchmark dataset.