Reading the Mood Behind Words: Integrating Prosody-Derived Emotional Context into Socially Responsive VR Agents

TL;DR

This paper introduces a VR interaction system that integrates prosody-based emotional cues into conversational agents, significantly enhancing dialogue naturalness and user engagement by leveraging real-time speech emotion recognition.

Contribution

It presents a novel pipeline combining speech emotion recognition with LLM-based dialogue to improve emotional congruence in VR agents, which was not previously explored.

Findings

Participants preferred the emotion-aware agent (93.3%)

Improved dialogue quality and naturalness

Enhanced user engagement and rapport

Abstract

In VR interactions with embodied conversational agents, users' emotional intent is often conveyed more by how something is said than by what is said. However, most VR agent pipelines rely on speech-to-text processing, discarding prosodic cues and often producing emotionally incongruent responses despite correct semantics. We propose an emotion-context-aware VR interaction pipeline that treats vocal emotion as explicit dialogue context in an LLM-based conversational agent. A real-time speech emotion recognition model infers users' emotional states from prosody, and the resulting emotion labels are injected into the agent's dialogue context to shape response tone and style. Results from a within-subjects VR study (N=30) show significant improvements in dialogue quality, naturalness, engagement, rapport, and human-likeness, with 93.3% of participants preferring the emotion-aware agent.