Emotion-Conditioned Short-Horizon Human Pose Forecasting with a Lightweight Predictive World Model

TL;DR

This paper explores using facial expression-derived emotion embeddings to improve short-term human pose prediction with a lightweight autoregressive model, showing that emotion signals can enhance prediction accuracy in emotion-driven sequences.

Contribution

It introduces a novel multimodal fusion approach using a gating mechanism within a lightweight predictive world model for emotion-conditioned pose forecasting.

Findings

Normalized gating fusion improves prediction accuracy for emotion-driven sequences.

Emotion embeddings act as auxiliary signals influencing pose prediction.

Counterfactual experiments show trajectory sensitivity to emotion input changes.

Abstract

Short-term human pose prediction plays a crucial role in interactive systems, assistive robots, and emotion-aware human-computer interaction[1-3]. While current trajectory prediction models primarily rely on geometric motion cues, they often overlook the underlying emotional signals influencing human motion dynamics[4-5]. This paper investigates whether facial expression-derived emotion embeddings can provide auxiliary conditional signals for short-term pose prediction. To further evaluate multimodal conditionation in a recursive prediction setting, we propose a lightweight autoregressive predictive world model that performs 15-step rolling pose prediction. This framework combines pose keypoints with emotion embeddings through a learnable gating mechanism and performs autoregressive unfolding prediction using a recurrent sequence model based on a two-layer LSTM architecture. Experiments were conducted on two small-scale pose-emotion video datasets: controlled motion sequences with minimal facial expression changes and, natural emotion-driven motion sequences with considerable facial expression changes. The results show that simple multimodal fusion does not consistently improve prediction accuracy, while normalized gating fusion significantly enhances the performance of emotion-driven motion sequences. Furthermore, counterfactual perturbation experiments demonstrate that the predicted trajectory exhibits measurable sensitivity to changes in multimodal input, suggesting that facial expression embeddings act as auxiliary conditional signals rather than redundant features. In summary, these results indicate that incorporating facial expression-derived emotion embeddings into emotion-conditional short-term pose prediction based on a lightweight predictive world model architecture is a feasible approach.