Reinforcing Trustworthiness in Multimodal Emotional Support Systems

TL;DR

This paper introduces MultiMood, a multimodal framework that combines video, audio, and text data with reinforcement learning to enhance trustworthiness and emotional support quality in AI systems, achieving state-of-the-art results.

Contribution

The paper presents a novel multimodal emotional support framework that integrates psychological criteria and reinforcement learning to improve trustworthiness and response quality.

Findings

MultiMood outperforms existing models on MESC and DFEW datasets.

Reinforcement learning enhances trustworthiness as validated by human and LLM evaluations.

The framework effectively combines multimodal data for empathetic, standards-aligned responses.

Abstract

In today's world, emotional support is increasingly essential, yet it remains challenging for both those seeking help and those offering it. Multimodal approaches to emotional support show great promise by integrating diverse data sources to provide empathetic, contextually relevant responses, fostering more effective interactions. However, current methods have notable limitations, often relying solely on text or converting other data types into text, or providing emotion recognition only, thus overlooking the full potential of multimodal inputs. Moreover, many studies prioritize response generation without accurately identifying critical emotional support elements or ensuring the reliability of outputs. To overcome these issues, we introduce \textsc{ MultiMood}, a new framework that (i) leverages multimodal embeddings from video, audio, and text to predict emotional components and to produce responses responses aligned with professional therapeutic standards. To improve trustworthiness, we (ii) incorporate novel psychological criteria and apply Reinforcement Learning (RL) to optimize large language models (LLMs) for consistent adherence to these standards. We also (iii) analyze several advanced LLMs to assess their multimodal emotional support capabilities. Experimental results show that MultiMood achieves state-of-the-art on MESC and DFEW datasets while RL-driven trustworthiness improvements are validated through human and LLM evaluations, demonstrating its superior capability in applying a multimodal framework in this domain.