Multimodal Hidden Markov Models for Persistent Emotional State Tracking

TL;DR

This paper introduces a lightweight multimodal hidden Markov model framework for tracking persistent emotional states in conversations, enhancing interpretability and efficiency over existing utterance-level emotion recognition methods.

Contribution

It proposes a novel sticky factorial HDP-HMM approach for modeling emotional regimes using multimodal data, improving interpretability and computational efficiency.

Findings

The model predicts more interpretable emotional regimes than Gaussian HMM.

Regimes can be reliably recovered from multimodal valence-arousal trajectories.

Using emotional regimes improves LLM response quality in clinical dialogue contexts.

Abstract

Tracking an interpretable emotional arc of a conversation via the sentiment of individual utterances processed as a whole is central to both understanding and guiding communication in applied, especially clinical, conversational contexts. Existing approaches to emotion recognition operate at the utterance level, obscuring the persistent phases that characterize real conversational dynamics. We propose a lightweight framework that models conversational emotion as a sequence of latent emotional regimes using sticky factorial HDP-HMMs over multimodal valence-arousal representations derived from simultaneous video, audio and textual input. We evaluate the quality of regime prediction using LLM-as-a-Judge, geometric, and temporal consistency metrics, demonstrating that the sticky HDP-HMM produces more interpretable regime sequences than the baseline Gaussian HMM at a fraction of the computational cost of LLM-based dialogue state tracking methods. In addition, Question-Answer experiments in a clinical dataset suggest that meaningful emotional phases can reliably be recovered from multimodal valence-arousal trajectories and used to improve the quality of LLM responses in unstable affective regimes via context augmentation. This framework thus opens a path toward interpretable, lightweight, and actionable analysis of conversational emotion dynamics at scale.