PaperToPlace: Transforming Instruction Documents into Spatialized and Context-Aware Mixed Reality Experiences

TL;DR

PaperToPlace transforms traditional paper instructions into spatialized mixed reality experiences, enhancing usability and reducing effort by leveraging authoring and consumption pipelines with machine learning for spatial placement.

Contribution

Introduces a novel workflow that converts paper instructions into MR experiences with spatial placement, improving interaction and comprehension.

Findings

Authoring pipeline is usable and effective with machine learning for spatial extraction.

Consumption pipeline reduces effort and improves instruction delivery in MR.

Participants preferred spatialized instructions for ease of use.

Abstract

While paper instructions are one of the mainstream medium for sharing knowledge, consuming such instructions and translating them into activities are inefficient due to the lack of connectivity with physical environment. We present PaperToPlace, a novel workflow comprising an authoring pipeline, which allows the authors to rapidly transform and spatialize existing paper instructions into MR experience, and a consumption pipeline, which computationally place each instruction step at an optimal location that is easy to read and do not occlude key interaction areas. Our evaluations of the authoring pipeline with 12 participants demonstrated the usability of our workflow and the effectiveness of using a machine learning based approach to help extracting the spatial locations associated with each steps. A second within-subject study with another 12 participants demonstrates the merits of our consumption pipeline by reducing efforts of context switching, delivering the segmented instruction steps and offering the hands-free affordances.