SketchMeHow: Interactive Projection Guided Task Instruction with User Sketches

TL;DR

SketchMeHow introduces an interactive system that uses user sketches for real-time task guidance, combining sketch analysis with projection technology to assist users in daily tasks effectively.

Contribution

It presents a novel sketch-based instruction framework integrating spatial augmented reality for real-time task guidance, which is a new approach in human-computer interaction.

Findings

Helps novice users complete tasks efficiently

Achieves high user satisfaction in case studies

Broadens sketch-based applications in real-world HCI

Abstract

In this work, we propose an interactive general instruction framework SketchMeHow to guidance the common users to complete the daily tasks in real-time. In contrast to the conventional augmented reality-based instruction systems, the proposed framework utilizes the user sketches as system inputs to acquire the users' production intentions from the drawing interfaces. Given the user sketches, the designated task instruction can be analyzed based on the sub-task division and spatial localization for each task. The projector-camera system is adopted in the projection guidance to the end-users with the spatial augmented reality technology. To verify the proposed framework, we conducted two case studies of domino arrangement and bento production. From our user studies, the proposed systems can help novice users complete the tasks efficiently with user satisfaction. We believe the proposed SketchMeHow can broaden the research topics in sketch-based real-world applications in human-computer interaction.