Adaptive Gen-AI Guidance in Virtual Reality: A Multimodal Exploration of Engagement in Neapolitan Pizza-Making

TL;DR

This study explores how multimodal behavioral metrics can evaluate user engagement in adaptive VR environments, revealing that moderate AI adaptivity optimally enhances engagement during a cultural culinary experience.

Contribution

It introduces a multimodal assessment framework for adaptive VR systems and demonstrates that moderate AI adaptivity improves user engagement in a cultural learning context.

Findings

Moderate adaptivity maximizes user engagement.

Balanced AI adaptivity reduces unnecessary exploration.

Visual attention increases with moderate adaptivity.

Abstract

Virtual reality (VR) offers promising opportunities for procedural learning, particularly in preserving intangible cultural heritage. Advances in generative artificial intelligence (Gen-AI) further enrich these experiences by enabling adaptive learning pathways. However, evaluating such adaptive systems using traditional temporal metrics remains challenging due to the inherent variability in Gen-AI response times. To address this, our study employs multimodal behavioural metrics, including visual attention, physical exploratory behaviour, and verbal interaction, to assess user engagement in an adaptive VR environment. In a controlled experiment with 54 participants, we compared three levels of adaptivity (high, moderate, and non-adaptive baseline) within a Neapolitan pizza-making VR experience. Results show that moderate adaptivity optimally enhances user engagement, significantly reducing unnecessary exploratory behaviour and increasing focused visual attention on the AI avatar. Our findings suggest that a balanced level of adaptive AI provides the most effective user support, offering practical design recommendations for future adaptive educational technologies.