Mutltimodal AI Companion for Interactive Fairytale Co-creation

TL;DR

This paper introduces AI.R Taletorium, a multimodal AI system that actively involves children in creating fairy tales through interactive storytelling and visualization, enhancing early childhood education and engagement.

Contribution

The work presents a novel multimodal AI companion with a bidirectional connection mechanism, enabling interactive fairy tale co-creation using text and doodles, which was not addressed in prior systems.

Findings

Generated meaningful and vivid fairy tales with limited training data.

Successfully completed full interaction cycle with various inputs.

Enhanced engagement in early childhood education through multimodal interaction.

Abstract

AI fairy tale companions play an important role in early childhood education as an augmentation for parents' efforts to close the participation gap and boost kids' mental and language development. Existing systems are generally designed to provide vivid materials as unidirectional entertaining reading environments, e.g, visualizing inputting texts. However, due to the limited vocabulary of kids, these systems failed to afford effective interaction to motivate kids to write their own fairy tales. In this work, we propose AI.R Taletorium, an illustrative, immersive, and inclusive multimodal AI companion, for interactive fairy tale co-creation that actively involves kids to create fairy tales with both the AI agent and their normal peers. AI.R Taletorium consists a neural story generator and a doodler-based fairy tale visualizer. We design a character-centric bidirectional connection mechanism between the story generator and visualizer equipped with Contrastive Language Image Pretraining (CLIP), thus enabling kids to participant in the story generation process by simple sketching. Extensive experiments and user studies show that our system was able to generate and visualize meaningful and vivid fairy tales with limited training data and complete the full interaction cycle under various inputs (text, doodler) through the bidirectional connection.