SketchPlay: Intuitive Creation of Physically Realistic VR Content with Gesture-Driven Sketching

TL;DR

SketchPlay is a VR framework that enables intuitive creation of physically realistic scenes through gesture-driven sketching, making complex content creation accessible for non-experts and enhancing expressiveness and user engagement.

Contribution

It introduces a novel VR interaction method combining sketches and gestures to generate dynamic, physically realistic content, simplifying the creation process for non-expert users.

Findings

Outperforms traditional text-driven methods in expressiveness.

Enhances user experience and engagement.

Enables creation of complex physical phenomena like cloth and elastic dynamics.

Abstract

Creating physically realistic content in VR often requires complex modeling tools or predefined 3D models, textures, and animations, which present significant barriers for non-expert users. In this paper, we propose SketchPlay, a novel VR interaction framework that transforms humans' air-drawn sketches and gestures into dynamic, physically realistic scenes, making content creation intuitive and playful like drawing. Specifically, sketches capture the structure and spatial arrangement of objects and scenes, while gestures convey physical cues such as velocity, direction, and force that define movement and behavior. By combining these complementary forms of input, SketchPlay captures both the structure and dynamics of user-created content, enabling the generation of a wide range of complex physical phenomena, such as rigid body motion, elastic deformation, and cloth dynamics. Experimental results demonstrate that, compared to traditional text-driven methods, SketchPlay offers significant advantages in expressiveness, and user experience. By providing an intuitive and engaging creation process, SketchPlay lowers the entry barrier for non-expert users and shows strong potential for applications in education, art, and immersive storytelling.