Dynamic Multimodal Expression Generation for LLM-Driven Pedagogical Agents: From User Experience Perspective

TL;DR

This paper introduces a novel LLM-driven multimodal expression generation method for pedagogical agents in VR, improving naturalness, engagement, and perceived effectiveness by dynamically aligning speech and gestures with instructional content.

Contribution

It presents a new dynamic multimodal expression generation approach for VR pedagogical agents, enhancing natural interaction and learner engagement compared to static methods.

Findings

Enhanced learner engagement and perceived learning effectiveness.

Reduced fatigue and boredom during VR learning.

Improved perceptions of human-likeness and social presence.

Abstract

In virtual reality (VR) educational scenarios, Pedagogical agents (PAs) enhance immersive learning through realistic appearances and interactive behaviors. However, most existing PAs rely on static speech and simple gestures. This limitation reduces their ability to dynamically adapt to the semantic context of instructional content. As a result, interactions often lack naturalness and effectiveness in the teaching process. To address this challenge, this study proposes a large language model (LLM)-driven multimodal expression generation method that constructs semantically sensitive prompts to generate coordinated speech and gesture instructions, enabling dynamic alignment between instructional semantics and multimodal expressive behaviors. A VR-based PA prototype was developed and evaluated through user experience-oriented subjective experiments. Results indicate that dynamically generated multimodal expressions significantly enhance learners' perceived learning effectiveness, engagement, and intention to use, while effectively alleviating feelings of fatigue and boredom during the learning process. Furthermore, the combined dynamic expression of speech and gestures notably enhances learners' perceptions of human-likeness and social presence. The findings provide new insights and design guidelines for building more immersive and naturally expressive intelligent PAs.