PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR

TL;DR

PlayWrite is a mixed-reality system enabling users to co-author narratives through direct manipulation and play in XR, with AI interpreting actions into structured story elements and generating final stories.

Contribution

This work introduces a novel multimodal, XR-based narrative co-authoring system that integrates direct manipulation, play, and AI to support creative storytelling.

Findings

Users found the system fostered improvisation and playfulness.

The AI acted as a collaborative partner, sparking new ideas.

Participants overcame creative blocks using the system.

Abstract

Current AI writing tools, which rely on text prompts, poorly support the spatial and interactive nature of storytelling where ideas emerge from direct manipulation and play. We present PlayWrite, a mixed-reality system where users author stories by directly manipulating virtual characters and props. A multi-agent AI pipeline interprets these actions into Intent Frames -structured narrative beats visualized as rearrangeable story marbles on a timeline. A large language model then transforms the user's assembled sequence into a final narrative. A user study (N=13) with writers from varying domains found that PlayWrite fosters a highly improvisational and playful process. Users treated the AI as a collaborative partner, using its unexpected responses to spark new ideas and overcome creative blocks. PlayWrite demonstrates an approach for co-creative systems that move beyond text to embrace direct manipulation and play as core interaction modalities.