Understanding Emotional Body Expressions via Large Language Models

TL;DR

This paper introduces EAI-LLM, a model that recognizes emotions from body movements and generates textual explanations by leveraging large language models and a novel skeleton tokenization approach, improving interpretability and accuracy.

Contribution

The paper presents a multi-granularity skeleton tokenizer and a unified skeleton token module that enable LLMs to recognize emotions from body data and generate explanations, addressing dataset heterogeneity and limited data.

Findings

Achieves recognition accuracy comparable to existing methods.

Generates detailed emotion descriptions using background knowledge from LLMs.

Performs well even with limited labeled skeleton data.

Abstract

Emotion recognition based on body movements is vital in human-computer interaction. However, existing emotion recognition methods predominantly focus on enhancing classification accuracy, often neglecting the provision of textual explanations to justify their classifications. In this paper, we propose an Emotion-Action Interpreter powered by Large Language Model (EAI-LLM), which not only recognizes emotions but also generates textual explanations by treating 3D body movement data as unique input tokens within large language models (LLMs). Specifically, we propose a multi-granularity skeleton tokenizer designed for LLMs, which separately extracts spatio-temporal tokens and semantic tokens from the skeleton data. This approach allows LLMs to generate more nuanced classification descriptions while maintaining robust classification performance. Furthermore, we treat the skeleton sequence as a specific language and propose a unified skeleton token module. This module leverages the extensive background knowledge and language processing capabilities of LLMs to address the challenges of joint training on heterogeneous datasets, thereby significantly enhancing recognition accuracy on individual datasets. Experimental results demonstrate that our model achieves recognition accuracy comparable to existing methods. More importantly, with the support of background knowledge from LLMs, our model can generate detailed emotion descriptions based on classification results, even when trained on a limited amount of labeled skeleton data.