DrawTalking: Building Interactive Worlds by Sketching and Speaking

TL;DR

DrawTalking enables users to create and control interactive worlds through sketching and speaking, offering a flexible, programming-like experience without coding, suitable for creative storytelling and exploration.

Contribution

It introduces a novel natural interface combining sketching and speech for interactive world-building without programming.

Findings

User engagement with the prototype was high.

The approach is applicable to diverse creative use cases.

It has potential to inspire future research in natural interfaces.

Abstract

We introduce DrawTalking, an approach to building and controlling interactive worlds by sketching and speaking while telling stories. It emphasizes user control and flexibility, and gives programming-like capability without requiring code. An early open-ended study with our prototype shows that the mechanics resonate and are applicable to many creative-exploratory use cases, with the potential to inspire and inform research in future natural interfaces for creative exploration and authoring.