An Exploratory Study on Multi-modal Generative AI in AR Storytelling

TL;DR

This paper explores how generative AI can enhance multi-modal content creation in AR storytelling, analyzing user preferences, interactions, and content quality to inform future design considerations.

Contribution

It provides an exploratory analysis of AI-generated content in AR storytelling, introduces a new design space, and develops a testbed for multi-modal content generation.

Findings

Participants preferred certain modalities for storytelling.

AI interactions influenced content creation preferences.

The quality of AI-generated content was evaluated positively.

Abstract

Storytelling in AR has gained attention due to its multi-modality and interactivity. However, generating multi-modal content for AR storytelling requires expertise and efforts for high-quality conveyance of the narrator's intention. Recently, Generative-AI (GenAI) has shown promising applications in multi-modal content generation. Despite the potential benefit, current research calls for validating the effect of AI-generated content (AIGC) in AR Storytelling. Therefore, we conducted an exploratory study to investigate the utilization of GenAI. Analyzing 223 AR videos, we identified a design space for multi-modal AR Storytelling. Based on the design space, we developed a testbed facilitating multi-modal content generation and atomic elements in AR Storytelling. Through two studies with N=30 experienced storytellers and live presenters, we 1. revealed participants' preferences for modalities, 2. evaluated the interactions with AI to generate content, and 3. assessed the quality of the AIGC for AR Storytelling. We further discussed design considerations for future AR Storytelling with GenAI.