Pose-based Body Language Recognition for Emotion and Psychiatric Symptom Interpretation

TL;DR

This paper presents a pose-based framework for recognizing emotions and psychiatric symptoms from RGB videos, designed to work with limited data and validated on public and specialized datasets.

Contribution

It introduces a novel two-stage system combining pose estimation and temporal analysis for emotion and psychiatric symptom recognition, with demonstrated transferability and effectiveness.

Findings

Outperforms existing methods on the URMC dataset

Effective with small training datasets

Validated on multiple public action recognition datasets

Abstract

Inspired by the human ability to infer emotions from body language, we propose an automated framework for body language based emotion recognition starting from regular RGB videos. In collaboration with psychologists, we further extend the framework for psychiatric symptom prediction. Because a specific application domain of the proposed framework may only supply a limited amount of data, the framework is designed to work on a small training set and possess a good transferability. The proposed system in the first stage generates sequences of body language predictions based on human poses estimated from input videos. In the second stage, the predicted sequences are fed into a temporal network for emotion interpretation and psychiatric symptom prediction. We first validate the accuracy and transferability of the proposed body language recognition method on several public action recognition datasets. We then evaluate the framework on a proposed URMC dataset, which consists of conversations between a standardized patient and a behavioral health professional, along with expert annotations of body language, emotions, and potential psychiatric symptoms. The proposed framework outperforms other methods on the URMC dataset.